chapters/Chapter 23 - Corpus Analysis with Text-Fabric.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://annotation.github.io/text-fabric/images/tf-small.png\">\n",
    "\n",
    "# Chapter 23 - Corpus Analysis with Text-Fabric\n",
    "\n",
    "If you work in particular areas of humanities, it may be the case that you work frequently\n",
    "with the same corpus or set of corpora. You may also want to annotate that corpus with\n",
    "features of interest for your research. In this case, it is helpful to have a tool that \n",
    "allows you to quickly and easily access your text corpus, as well as to easily add features\n",
    "to certain words, phrases, sentences, etc. that can be retrieved later with simple queries.\n",
    "\n",
    "[Text-Fabric](https://annotation.github.io/text-fabric/) is a tool build specifically with\n",
    "these use-cases in mind. In this notebook, we will explore the basic data model and functionality of Text-Fabric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to model a text?\n",
    "\n",
    "There are lots of ways to store and model a text corpus. XML, for instance, \n",
    "is a hierarchical structure that allows for tree-like embedding of linguistic objects. \n",
    "It is the basis for standards such as TEI. But XML is not good at representing discontinuous\n",
    "items, something we encounter frequently in language:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = 'This sentence which I wrote is here.'\n",
    "\n",
    "# we can build such a tree like this:\n",
    "tree_hierarchy = \"\"\"\n",
    "\n",
    "<sentence>\n",
    "    <clause>\n",
    "    This sentence\n",
    "        <clause>\n",
    "        which I wrote\n",
    "        </clause>\n",
    "    is here.\n",
    "    </clause>\n",
    "</sentence>\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "# but how do we model this??\n",
    "sentence2 = 'This sentence—yes, this one—is here'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**XML is a form of in-line markup, where the annotations are mixed in with the text**. \n",
    "This makes it not only difficult to model discontinuous items, but also difficult to\n",
    "quickly select only those items of interest, since one has to first iterate over all\n",
    "of the annotation.\n",
    "\n",
    "**Text-Fabric is a form of stand-off markup, where the annotations are stored separately\n",
    "from the text**. This approach makes it easy to model dicontinuous items. It also maintains\n",
    "a separation of concerns between the text and the annotation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# stand-off markup in Text-Fabric style\n",
    "\n",
    "#          word number    1     2      3    4    5   6   7\n",
    "interrupted_sentence = 'This sentence—yes, this one—is here.'\n",
    "\n",
    "# word mapping to our new sentence objects\n",
    "# each number corresponds to a word in the text:\n",
    "\n",
    "object_to_slots = {\n",
    "    'sentence1': (1, 2, 6, 7),\n",
    "    'sentence2': (3, 4, 5),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we've defined our sentences in terms of the atomic items, or slots, they contain. \n",
    "In this case, we have arbitrarily chosen a \"word\" as the slots. But we could have chosen\n",
    "letters if we wanted. Notice that dividing the text this way allows the discontinuity to \n",
    "be reflected, without losing the information that sentence2 sits between the items of \n",
    "sentence 1.\n",
    "\n",
    "We can also assign integers to the new sentences themselves. That will allow us to associate\n",
    "features with those sentences. \n",
    "\n",
    "We need unique ID's then for each sentence. **Let's arbitrarily begin the sentence ID count at the \n",
    "number of slots in the corpus +1**. In this case, we have 7 words, so we'll say that ID 8 is the first\n",
    "sentence, ID 9 is the second, and so on. If we had more slots than 7, we would start the count higher \n",
    "of course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#          word number    1     2      3    4    5   6   7\n",
    "interrupted_sentence = 'This sentence—yes, this one—is here.'\n",
    "\n",
    "# word mapping to our new sentence objects\n",
    "# each number corresponds to a word in the text:\n",
    "\n",
    "object_to_slots = {\n",
    "    8: (1, 2, 6, 7),\n",
    "    9: (3, 4, 5),\n",
    "}\n",
    "\n",
    "# to keep track of which integers belong to which\n",
    "# kinds of objects, we also create a dictionary that \n",
    "# stores each integer's object type\n",
    "\n",
    "object_types = {\n",
    "    1: 'word',\n",
    "    2: 'word',\n",
    "    3: 'word',\n",
    "    4: 'word',\n",
    "    5: 'word',\n",
    "    6: 'word',\n",
    "    7: 'word',\n",
    "    8: 'sentence',\n",
    "    9: 'sentence',\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've divided the text into objects, we can begin to\n",
    "associate features with those objects. We've already seen an example of \n",
    "such a feature dictionary above with `object_types`. We can follow the\n",
    "same logic to assign other features. \n",
    "\n",
    "**The surface text itself can be modeled as a feature**. Done in this way,\n",
    "we can even choose to arbitrarily assign various kinds of transcriptions or\n",
    "formats to the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plain_text = {\n",
    "    1: 'This ',\n",
    "    2: 'sentence—',\n",
    "    3: 'yes, ',\n",
    "    4: 'this ',\n",
    "    5: 'one—',\n",
    "    6: 'is ',\n",
    "    7: 'here.'\n",
    "}\n",
    "\n",
    "plain_text_no_punctuation = {\n",
    "    1: 'This',\n",
    "    2: 'sentence',\n",
    "    3: 'yes',\n",
    "    4: 'this',\n",
    "    5: 'one',\n",
    "    6: 'is',\n",
    "    7: 'here'\n",
    "}\n",
    "punctuation = {\n",
    "    1: ' ',\n",
    "    2: '—',\n",
    "    3: ', ',\n",
    "    4: ' ',\n",
    "    5: '—',\n",
    "    6: ' ',\n",
    "    7: '.'\n",
    "}\n",
    "greek_transcription = {\n",
    "    1: 'Θις ',\n",
    "    2: 'σεντενσε—',\n",
    "    3: 'ιες, ',\n",
    "    4: 'θις ',\n",
    "    5: 'ωνε—',\n",
    "    6: 'ις ',\n",
    "    7: 'ἑρε.'\n",
    "}\n",
    "parts_of_speech = {\n",
    "    1: 'demonstrative',\n",
    "    2: 'noun',\n",
    "    3: 'exclamation',\n",
    "    4: 'demonstrative',\n",
    "    5: 'noun',\n",
    "    6: 'verb',\n",
    "    7: 'adverb',\n",
    "}\n",
    "sentence_types = {\n",
    "    8: 'normal',\n",
    "    9: 'interrupting',\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, the possibilities are endless. Hopefully the benefits of working this way are clear.\n",
    "If you're a native Greek speaker, perhaps you'd like to use the Greek transcription. In that case,\n",
    "you don't even need to load the normal text. Or perhaps you're not interested in punctuation, you\n",
    "can decide to load the text without punctuation. If this were an XML dataset, we'd have to iterate\n",
    "through every single tag and piece of associated data to clean out what we want. With the Text-Fabric\n",
    "model, we only do that once: when we construct the text. And from then on the data is ready to go."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text-Fabric Data Model \n",
    "\n",
    "The presentation above shows the core operating principle behind Text-Fabric,\n",
    "though some of the terms of different:\n",
    "\n",
    "| | Basic Components |\n",
    "|------|------------------------------------------------------------------------------------|\n",
    "| slot | sequential, atomic item used as a reference point for building the text (word or letter, etc.) |\n",
    "| node | A unique \"ID\" number that corresponds to either a slot, or an item defined by a range of slots. |\n",
    "| edge | a relationship between two nodes                                                   |\n",
    "| feature | data associated with a given node |\n",
    "\n",
    "With these components, we can model and represent a whole text. Text-Fabric handles all of the administration\n",
    "for us by writing files which contain all of the relevant mappings and information. To build a Text-Fabric\n",
    "corpus, we just have to feed the Text-Fabric Python module the right set of dictionaries.\n",
    "\n",
    "## Getting Ready-made Corpora with Text-Fabric\n",
    "You can use Text-Fabric with a TF dataset stored locally on your machine. But Text-Fabric\n",
    "also has the capability of retrieving ready-made corpora stored in its public Github\n",
    "repository. Here are the set of corpora that are currently stored in the repository:\n",
    "\n",
    "| acronym       | language/writing system | name                             | period           | description                                                       | converted by                         |\n",
    "|---------------|-------------------------|----------------------------------|------------------|-------------------------------------------------------------------|--------------------------------------|\n",
    "| athenaeus     | Greek                   | Works of Athenaeus               | 80 - 170         | Deipnosophistae                                                   | Ernst Boogert                        |\n",
    "| banks         | modern english          | Iain M. Banks                    | 1984 - 1987      | 99 words from the SF novel Consider Phlebas                       | Dirk Roorda                          |\n",
    "| bhsa          | Hebrew                  | Hebrew Bible                     | 1000 BC - 900 AD | Biblia Hebraica Stuttgartensia (Amstelodamensis)                  | Dirk Roorda + ETCBC                  |\n",
    "| dss           | Hebrew                  | Dead Sea Scrolls                 | 300 BC - 100 AD  | Transcriptions with morphology based on Martin Abegg's data files | Dirk Roorda, Jarod Jacobs            |\n",
    "| nena          | Aramaic                 | North Eastern Neo-Aramaic Corpus | 2000-on          | Nena Cambridge                                                    | Cody Kingham                         |\n",
    "| oldbabylonian | Akkadian / cuneiform    | Old Babylonian letters           | 1900 - 1600 BC   | Altbabylonische Briefe in Umschrift und Übersetzung               | Dirk Roorda, Cale Johnson            |\n",
    "| peshitta      | Syriac                  | Syriac Old Testament             | 1000 BC - 900 AD | Vetus Testamentum Syriace                                         | Dirk Roorda, Hannes Vlaardingerbroek |\n",
    "| quran         | Arabic                  | Quran                            | 600 - 900        | Quranic Arabic Corpus                                             | Dirk Roorda, Cornelis van Lit        |\n",
    "| syrnt         | Syriac                  | Syriac New Testament             | 0 - 1000         | Novum Testamentum Syriace                                         | Dirk Roorda, Hannes Vlaardingerbroek |\n",
    "| tisch         | Greek                   | New Testament                    | 50 - 450         | Greek New Testament in Tischendorf 8thEdition                     | Cody Kingham                         |\n",
    "| uruk          | proto-cuneiform         | Uruk                             | 4000 - 3100 BC   | Archaic tablets from Uruk                                         | Dirk Roorda, Cale Johnson            |\n",
    "\n",
    "See the links to all the corpora [here](https://annotation.github.io/text-fabric/About/Corpora/). \n",
    "Most of the ready-made corpora are currently ancient languages, but the Text-Fabric library is \n",
    "always expanding.\n",
    "\n",
    "We will load Text-Fabric now and begin working with the Syriac New Testament."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you don't have Text-Fabric installed,\n",
    "# uncomment below and run, and it should work\n",
    "\n",
    "#! pip install text-fabric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf.app import use\n",
    "\n",
    "syrnt = use('syrnt') # load the corpus using the ready-made method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `tf` Data Location\n",
    "\n",
    "Text-Fabric has now downloaded the corpus to your machine and loaded it into memory.\n",
    "\n",
    "**Note the links above lead to documentation about the features particular to this corpus.**\n",
    "Text-Fabric does not prescribe any kind of features. A feature is arbitrarilly created for\n",
    "a given corpus. So this documentation is important for knowing how to interact with that corpus.\n",
    "\n",
    "Let's find out where this data is loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(syrnt.mLocations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also see the version of the dataset we've loaded below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "syrnt.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The file path above + the version tells you where the Text-Fabric data files\n",
    "have been downloaded.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_versions = syrnt.mLocations[0]\n",
    "version = '/' + syrnt.version\n",
    "path_to_data = path_to_versions + version\n",
    "\n",
    "print(path_to_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF Data Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the files that have been \n",
    "downloaded. \n",
    "\n",
    "We'll use a set of terminal commands to peek into that \n",
    "folder. *Don't worry if you don't understand the terminal commands.*\n",
    "\n",
    "You could also navigate manually to the location indicated by the file path\n",
    "and have a look yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! ls $path_to_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### otype (\"object type\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see a bunch of files with the `tf` extension. These are all files \n",
    "which contain Text-Fabric formatted data. Let's have a look at the file\n",
    "called `otype`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! cat $path_to_data/otype.tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the entire contents of the file. The top of a TF file contains \n",
    "metadata tags, marked with `@` symbols, followed by the name of the field,\n",
    "and the metadata.\n",
    "\n",
    "Below the metadata we can see the actual data itself. The first column\n",
    "contains a range of node ID numbers, and the second column contains a \n",
    "string associated with those numbers, telling us—in this case—the \"otype\"\n",
    "or object type of each of these nodes.\n",
    "\n",
    "**Can you guess which of these items is the slots?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### oslots (\"object slots\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above in our mock dataset we mapped tuples of slots to sentence\n",
    "IDs. The `oslots` file does something similar to this, mapping a \n",
    "given node number to a range of corresponding slots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! head -20 $path_to_data/oslots.tf; echo \"\\n-- first 20 lines of file --\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the file begins with one column, but then switches to a single column.\n",
    "**That is because the Text-Fabric data format optimizes storage by *inferring* the node numbers\n",
    "when one node follows subsequently from another.** \n",
    "\n",
    "For example, above the first line, \n",
    "\n",
    "```\n",
    "109641\t1-13979\n",
    "``` \n",
    "\n",
    "defines the range of slots belonging to node `109641`. The second line:\n",
    "\n",
    "```\n",
    "13980-22772\n",
    "```\n",
    "\n",
    "Defines the range of slots belonging to `109642`, which is inferred from\n",
    "the fact it immediately follows `109642`. If there is interruption from one\n",
    "node to the next (i.e. they are not subsequent) then the second column will \n",
    "briefly re-appear to update the position. \n",
    "\n",
    "Note also that we can look up the object type of `109641` in the `otype` file\n",
    "referred to further above. It is a book node."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### feature files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at a different file. word.tf, which we can see from the \n",
    "[documentation](https://github.com/etcbc/syrnt/blob/master/docs/transcription-0.1.md#node-type-word)\n",
    "contains the plain text of a word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! head -25 $path_to_data/word.tf; echo \"\\n-- first 25 lines of file --\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file contains the plain text features corresponding to word nodes.\n",
    "\n",
    "If the first item in the file is associated with node 1, there is no need\n",
    "to write \"1\". Text-Fabric can simply assume the count."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interacting with a TF corpus\n",
    "\n",
    "Text-Fabric is not just a data model, but also a Python library that interacts with\n",
    "that model in efficient ways. The library supplies a set of Python objects that can\n",
    "be used to iterate over nodes, select features on those nodes, or compare relations\n",
    "between the nodes. \n",
    "\n",
    "The Text-Fabric objects are made available through the `api`.\n",
    "\n",
    "#### [Read about the Text-Fabric API here](https://annotation.github.io/text-fabric/Api/Fabric/#text-fabric-api)\n",
    "\n",
    "Here are the basic objects for interacting with a loaded Text-Fabric corpus:\n",
    "\n",
    "| object | what it does                          |\n",
    "|--------|---------------------------------------|\n",
    "| N      | gives access to nodes                 |\n",
    "| F      | gives access to node features         |\n",
    "| L      | retrieves embedding or embedded nodes |\n",
    "| T      | retrieves text and section markers    |\n",
    "| E      | retrieves edge data from a node       |\n",
    "\n",
    "Read about [other objects here](https://annotation.github.io/text-fabric/Api/Fabric/#loading)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dir(syrnt.api)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access all nodes\n",
    "\n",
    "Let's start off with simple node interactions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = syrnt.api.N"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_nodes = 0\n",
    "\n",
    "for node in N():\n",
    "    number_of_nodes += 1\n",
    "    \n",
    "print(number_of_nodes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've now iterated through all of the nodes in the corpus and counted them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access features of nodes\n",
    "\n",
    "If we want to know the `otype` (object type) of each node, we need the \n",
    "`F` (feature) object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "F = syrnt.api.F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [See F documentation](https://annotation.github.io/text-fabric/Api/Features/)\n",
    "\n",
    "We can get a feature value with the following pattern:\n",
    "\n",
    "```\n",
    "\n",
    "F.feature_name.v(node)\n",
    "\n",
    "```\n",
    "\n",
    "Remember that `1` will be the first slot in the corpus. Let's call the \n",
    "feature `otype` on this slot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "F.otype.v(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's apply this method to our count and count object types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_nodes = 0\n",
    "otype_counts = {}\n",
    "\n",
    "for node in N():\n",
    "    \n",
    "    # count the node and get the type\n",
    "    number_of_nodes += 1\n",
    "    otype = F.otype.v(node)\n",
    "    \n",
    "    # count the otype in the dictionary\n",
    "    if otype in otype_counts:\n",
    "        otype_counts[otype] += 1\n",
    "    else:\n",
    "        otype_counts[otype] = 1\n",
    "    \n",
    "print(number_of_nodes)\n",
    "\n",
    "print()\n",
    "for otype,count in otype_counts.items():\n",
    "    print(otype, 'count is', count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get a generator for specific feature types\n",
    "\n",
    "What if we're only interested in, say, the word nodes? \n",
    "\n",
    "We can use the following syntax to get a generator object (which we can\n",
    "convert to a list or simply loop over) that yields only certain feature types.\n",
    "\n",
    "```\n",
    "F.feature_name.s('feature_value')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_counts = 0\n",
    "\n",
    "for word in F.otype.s('word'):\n",
    "    word_counts += 1\n",
    "    \n",
    "print(word_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also do this for other features. Below we only iterate over the nodes\n",
    "that have a part of speech feature (sp) of `verb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "verb_count = 0\n",
    "\n",
    "for word in F.sp.s('verb'):\n",
    "    verb_count += 1\n",
    "\n",
    "print(verb_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access embedders or embedded nodes\n",
    "\n",
    "As we saw above in the `oslots` file, the first book node in the corpus\n",
    "contains a large range of slots:\n",
    "\n",
    "```\n",
    "109641    1-13979\n",
    "```\n",
    "\n",
    "But there are also nodes contained within the book that contain smaller\n",
    "ranges of slots, such as chapters.\n",
    "\n",
    "For instance, the first chapter in this corpus contains slots 1-290. In \n",
    "other words, according to the slots, the chapter is embedded in the book.\n",
    "\n",
    "**If a node's range of slots intersects with, and is smaller than, another\n",
    "node's, it is embedded in that node; and if it intersects and is bigger than\n",
    "another node's, it embeds that node**.\n",
    "\n",
    "For a more expansive definition, see the documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access embedded nodes using Text-Fabric's `L` object using the following syntax:\n",
    "\n",
    "#### [See L documentation](https://annotation.github.io/text-fabric/Api/Locality/#locality)\n",
    "\n",
    "```\n",
    "L.u(node, \"embedder_otype\")\n",
    "```\n",
    "or\n",
    "```\n",
    "L.d(node, \"embedded_otype\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "L = syrnt.api.L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_book_chapters = L.d(109641, 'chapter')\n",
    "\n",
    "print(first_book_chapters)\n",
    "print()\n",
    "print('first book contains', len(first_book_chapters), 'chapters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's retrieve the first chapter, starting with a slot this time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_chapter = L.u(1, 'chapter')\n",
    "\n",
    "first_chapter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that even when an item is embedded by only 1 node, `L` still\n",
    "returns a tuple. So we need to index the tuple to retrieve the node number\n",
    "of the first chapter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_chapter_node = first_chapter[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can go back down to the words, confirming that slot `1` is contained in this chapter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(L.d(first_chapter_node, 'word'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get text for a node\n",
    "\n",
    "The `T` object provides a convenient way to quickly access the surface text \n",
    "of any given node in the dataset.\n",
    "\n",
    "### [See the T documentation](https://annotation.github.io/text-fabric/Api/Text/)\n",
    "\n",
    "```\n",
    "T.text(node, fmt=\"format_here\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `fmt` is an optional argument, and the string value that accompanies it will\n",
    "be unique to the particular corpus you're working with. It allows you to specific alternative\n",
    "representations of the text such as transcriptions. You can also ignore it if you\n",
    "only want the default representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "T = syrnt.api.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(T.text(1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_verse = L.u(1, 'verse')[0]\n",
    "\n",
    "print(T.text(first_verse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(T.text(first_chapter_node))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additional formats are defined in the `otext` file with a series of metadata statements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! cat $path_to_data/otext.tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The relevant lines here are those beginning with `@fmt`. Note the names `lex-orig-full` and `lex-trans-full`. \n",
    "The subsequent values `{lexeme}` and `{lexeme_etcbc}` tell which node features are used to compile\n",
    "the text format. \n",
    "\n",
    "### [You can read about how a fmt string should be written here](https://annotation.github.io/text-fabric/Api/Text/#text-representation)\n",
    "\n",
    "Let's have a look at the two alternative formats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "T.text(first_chapter_node, fmt='lex-orig-full')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look closely and you'll notice this text is different from the one above, written now using only lexical forms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "T.text(first_chapter_node, fmt='lex-trans-full')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an ASCII trasncription version of the text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Section Data with T\n",
    "\n",
    "T can also be used to go back and forth from various section data to\n",
    "various nodes. \n",
    "\n",
    "\n",
    "### To go from a section to a node\n",
    "\n",
    "```\n",
    "T.nodeFromSection((section1, section1.1, section1.2))\n",
    "```\n",
    "\n",
    "Let's say we are interested in a given book, in this case the book \n",
    "of \"Hebrews\". We can select the book simply wiht `T.nodeFromSection`\n",
    "by feeding it a tuple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hebrews = T.nodeFromSection(('Hebrews',))\n",
    "\n",
    "print(hebrews)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can further specify chapter and verse like this:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hebrews_1_1 = T.nodeFromSection(('Hebrews', 1, 1))\n",
    "\n",
    "print(hebrews_1_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### To go from a node to a section\n",
    "\n",
    "```\n",
    "T.sectionFromNode(node)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want the section data from a node instead, we can use this function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_slot = 21342\n",
    "\n",
    "print(T.sectionFromNode(random_slot))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pretty representation of objects\n",
    "\n",
    "Text-Fabric also can represent nodes with formatted HTML to facilitate \n",
    "the data exploration process. \n",
    "\n",
    "### [Read about pretty methods here](https://annotation.github.io/text-fabric/apidocs/html/tf/applib/display.html#tf.applib.display.plain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pretty = syrnt.pretty\n",
    "plain = syrnt.plain\n",
    "prettyTuple = syrnt.prettyTuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pretty(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pretty(first_verse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prettyTuple((first_verse, 1), seq=0) # seq = result number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that with `prettyTuple`, we get highlighting behavior when an \n",
    "embedded node is included in the tuple."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Writing Text-Fabric Queries\n",
    "\n",
    "You can choose to maneuver the corpus programmatically with Python classes as we've \n",
    "seen above. But you can also use the `Text-Fabric` query language, Search, which is useful\n",
    "for quickly writing patterns that express relationships within the text.\n",
    "\n",
    "Note that TF Search has its own syntax, which you can read about below.\n",
    "\n",
    "### [Read about the syntax for TF Search](https://annotation.github.io/text-fabric/Use/Search/)\n",
    "\n",
    "The syntax is relatively straightforward: we specify nodes or features of those \n",
    "nodes, and we can specific embedding relations by indenting one node underneath another.\n",
    "And we can specify particular sequences between nodes embedded at the same level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search = syrnt.search # NB: not stored under standard API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "verb_query = search('word sp=verb') # query in the string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(verb_query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "verb_query[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the results of the search is returned as tuples of node numbers.\n",
    "\n",
    "We can take these nodes and do normal Text-Fabric things with them, as I showed\n",
    "above.\n",
    "\n",
    "We can also visualize the query results with the show method, which applies the\n",
    "pretty methods to a list of tuples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show = syrnt.show"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show(verb_query[:3]) # show first 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can also write more advanced queries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write the query in a large string\n",
    "# indentation specifies embedding into verse\n",
    "# <: specifies adjacent order between the two words\n",
    "noun_verb_query = \"\"\"\n",
    "\n",
    "verse\n",
    "    word sp=noun\n",
    "    <: word sp=verb\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "noun_verb_results = search(noun_verb_query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we can also use the optional argument `end` to restrict the `show` method to a certain \n",
    "number of results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show(noun_verb_results, end=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a Text-Fabric Corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have been working with a ready-made Text-Fabric Corpus. But you can also\n",
    "build your own with whatever text you already have. Text-Fabric also has \n",
    "a class for compiling and saving Text-Fabric files. \n",
    "\n",
    "### [See the tutorial to make your own corpus](https://nbviewer.jupyter.org/github/annotation/banks/blob/master/programs/convert.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding Corpus Annotations\n",
    "\n",
    "It is very easy to add annotations on top of an existing corpus. All you need to do is\n",
    "link the annotations to a given node number. \n",
    "\n",
    "If you are introducing manual annotations, for instance, you can export a .csv file where\n",
    "each row corresponds with a given node number. Then you can manually add your features to\n",
    "the nodes and import them back into Python to save in Text-Fabric format. \n",
    "\n",
    "\n",
    "Let's try an example right now. We're going to annotate just the first 5 words of the corpus\n",
    "with random data and save it to this folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# random annotations on first 5 nodes / slots of the corpus\n",
    "\n",
    "my_annotations = [\n",
    "    [1, 'X'],\n",
    "    [2, 'S'],\n",
    "    [3, 'J'],\n",
    "    [4, 'X'],\n",
    "    [5, 'S'],\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put annotations into a feature dictionary\n",
    "# where each key is a node number, and each\n",
    "# value is the feature value of the annotation\n",
    "\n",
    "annotation_features = {}\n",
    "\n",
    "for node, feature in my_annotations:\n",
    "    annotation_features[node] = feature\n",
    "    \n",
    "print(annotation_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# place the feature dictionary into another dictionary where the \n",
    "# key is the name of the feature and the value is the dictionary itself\n",
    "\n",
    "feature_dict = {'random_annotation': annotation_features}\n",
    "\n",
    "# Create a new dictionary that contains relevant metadata for your annotation feature\n",
    "# you also need to specify the @valueType of your feature, which should either be int or string\n",
    "metadata_dict = {\n",
    "    'random_annotation': {\n",
    "        'description': 'this is a random annotation',\n",
    "        'valueType': 'str' # obligatory\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can export the annotation features. We initialize a new Text-Fabric object and \n",
    "specify the location we want the new .tf files to be saved to.\n",
    "\n",
    "We will save the new feature to this directory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf.fabric import Fabric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TF = Fabric(locations='./')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The export happens below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TF.save(nodeFeatures=feature_dict, metaData=metadata_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tf` file has now been added. We can see it below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! cat random_annotation.tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can load our feature back into Text-Fabric if we'd like to use it for analysis. \n",
    "\n",
    "Since we're loading data from this directory as well as the default one, we'll use a slightly\n",
    "different approach to loading Text-Fabric.\n",
    "\n",
    "**You can use this load method anytime you need to load additional TF files from alternative locations**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tf.app import use\n",
    "\n",
    "# load TF data, give the default data path \n",
    "# as well as this current directory to the locations argument\n",
    "# put the paths in a list\n",
    "TF = Fabric(locations=[path_to_data, './'])\n",
    "\n",
    "# load all features\n",
    "# alternatively, you could call TF.load()\n",
    "# and give it a string of features you want to load\n",
    "API = TF.loadAll()\n",
    "\n",
    "# now reload the syrnt, but now give it the\n",
    "# pre-loaded API with our customized dataset\n",
    "syrnt = use('syrnt', api=API)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "F = syrnt.api.F\n",
    "\n",
    "annotated_nodes = [1, 2, 3, 4, 5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for node in annotated_nodes:\n",
    "    print(node, F.random_annotation.v(node))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contribute to Text-Fabric\n",
    "\n",
    "Text-Fabric is an ongoing project, with the goal of simplifying\n",
    "and streamlining research workflows while helping new programers apply corpus\n",
    "linguistic methods to their work. If you're interested in joining our community,\n",
    "follow the join link below and become a part of the Ancient-Data Slack (not just Ancient data anymore!)\n",
    "\n",
    "[join Ancient-Data Slack](https://join.slack.com/t/ancient-data/shared_invite/zt-bf37fg1v-b8nZt278EM3a30wjSTYxBQ)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}