Visualizing_Association_Rules.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization\n",
    "\n",
    "Visualization is successfully used to communicate both abstract and concrete ideas. The application of visualization falls\n",
    "into two phases. First, the exploration phase where the analysts will use graphics that are mostly incompatible for presentation purposes but make it easy to find interesting and important features of the data. The amount of interaction needed during exploration is very high and includes filtering, zooming and rearranging data. After key findings are discovered in the\n",
    "data, these findings must be presented in a way suitable for presentation for a larger audience. In this second phase it is important that the analyst can manipulate the presentation to clearly highlight the findings.\n",
    "\n",
    "### arulesViz\n",
    "\n",
    "This R package is a simple unified interface for many kinds of existing plots to visualize data. **`arulesViz`** package will automatically loads other needed packages like arules for handling and mining association rules. In this notebook we load the “Groceries” data set which is included in arules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# install.packages(\"arulesViz\",repo=\"http://cran.stat.ucla.edu/\")\n",
    "library(\"arulesViz\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data(\"Groceries\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Groceries contains sales data from a local grocery store with 9835 transactions and 169 items (product groups). The summary shows some basic statistics of the data set. For example, that the data set is rather sparse with a density just above 2.6%, that “whole milk” is the most popular item and that the average transaction contains less than 5 items.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "summary(Groceries)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate association rules with a suppor of 0.001 and confidence of 0.5 using Apriori algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rules <- apriori(Groceries, parameter=list(support=0.001, confidence=0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a set of 5668 association rules. The top three rules with respect to the lift measure are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "inspect(head(sort(rules, by =\"lift\"),3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is not feasible to go through all the 5668 rules manually as it is not a viable option. Explore the rules using different visualization techniques implemented in arulesViz. All implemented visualization techniques share the following interface:\n",
    "\n",
    "$$ plot(x, method = NULL, measure = \"support\", shading = \"lift\", interactive = FALSE, data = NULL, control = NULL, ...) $$\n",
    "\n",
    "where x is the set of rules to be visualized, method is the visualization method, measure and shading contain the interest measures used by the plot, interactive indicates if we want to interactively explore or just present the rules, data can contain the transaction data set used to mine the rules (only necessary for some methods) and control is a list with further control arguments to customize the plot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### scatter plot\n",
    "\n",
    "scatter plot will have support and confidence on the axes. In addition a third measure by default lift is used as the color (gray level) of the points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plot(rules, method = \"scatter\", measure = \"support\", shading = \"lift\", interactive = FALSE, data = Groceries,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot suggests rules with high confidence have typically a relatively low support. Customize the plot by switching lift and confidence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plot(rules, measure=c(\"support\", \"lift\"), shading=\"confidence\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is easy to identify all rules with high lift in above plot. Below shown is a special version of a scatter plot called Twokey\n",
    "plot. Here support and confidence are used for the x and y-axes and the color of the points is used to indicate “order,” i.e., the number of items contained in the rule. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plot(rules, shading=\"order\", control=list(main = \"Two-key plot\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the plot it is clear that order and support have a very strong inverse relationship, which is a known fact for association rules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Matrix-based visualizations\n",
    "\n",
    "Matrix-based visualization techniques organize the antecedent and consequent itemsets on the x and y-axes, respectively. A selected interest measure is displayed at the intersection of the antecedent and consequent of a given rule. If no rule is available for a antecedent/consequent combination the intersection area is left blank.\n",
    "\n",
    "The visualized matrix is constructed as follows. We start with the set of association rules. \n",
    "\n",
    "$$ R = {\\{<a_1, c_1, m_1>, . . .<a_i, c_i, m_i>, . . .<a_n, c_n, m_n>\\}} $$\n",
    "\n",
    "where $a_i$ is the antecedent, $c_i$ is the consequent and $m_i$ is the selected interest measure for the i-th rule for i = 1, . . . , n. The set of K unique antecedents and L unique consequent are identified. Create a **L × K** matrix **`M`** with one column for each unique antecedent and one row for each unique consequent. Finally, populate the matrix by setting $M_lk$ = $m_i$ for i = 1, . . . , n and _l_ and _k_ corresponding to the position of $a_i$ and $c_i$ in the matrix. M will contain many empty cells since many potential association rules will not meet the required minimum thresholds on support and confidence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "subrules <- rules[quality(rules)$confidence > 0.8]\n",
    "subrules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plot(subrules, method=\"matrix\", measure=\"lift\",cex.names=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since there is not much space for long labels in the plot, only numbers are shown as labels for rows and columns. Complete itemsets are printed for look-up. The rows and columns in the matrix can be reordered such that rules with similar values of the interest measure are presented closer together. This removes some of the fragmentation in the matrix display and therefore makes it easier to see structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plot(subrules, method=\"matrix\", measure=\"lift\", control=list(reorder=TRUE))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we see the emergence of two large blocks of rules with two different consequents and then smaller blocks for the rest. \n",
    "\n",
    "There are many ways available for ordering the multivariate data. Typically, finding the optimal or der subject to some defined objective function is a difficult combinatorial optimizationproblem which involves for n objects to check many of the O(n!) possible permutations. Listed below are the different methods and the objective function they try to optimize. \n",
    "\n",
    "**PCA** – First principal component. Uses the first principal component for the data matrix and for the transposed matrix as order for rows and columns. This is a very fast approach which avoids the expensive distance matrix computation.\n",
    "\n",
    "**TSP** – Traveling salesperson problem solver. Computes distance matrices between rows and between columns and solves two separate TSPs. By default the nearest insertion heuristic is used. This method is reasonably fast for small rule sets, but the distance matrix computation and the associated memory requirements make it impractical for larger sets.\n",
    "\n",
    "**HC** – Hierarchical clustering. Computes distance matrices for rows and columns and then clusters twice. Distance matrix computation is again limiting the approach for smaller sets.\n",
    "\n",
    "**max, avg and median** – Reorder rows/columns by their maximum, average or median value. This extremely simple and fast methods provides sometimes good visualizations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we specify a vector with two measures, both measures are used simultaneously using color hue for one measure and luminance and chroma together for the other. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plot(subrules, method=\"matrix\", measure=c(\"lift\", \"confidence\"), control=list(reorder=TRUE))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grouped matrix-based visualization\n",
    "\n",
    "Matrix-based visualization is limited in the number of rules it can visualize effectively since large sets of rules typically also have large sets of unique antecedents/consequents. This matrix-based visualization uses grouping of rules via clustering to handle a larger number of rules.  Grouped rules are presented as an aggregate in the matrix. \n",
    "\n",
    "A direct approach to cluster itemsets is to define a distance metric between two itemsets $X_i$ and $X_j$ . A good choice is the Jaccard distance defined as\n",
    "\n",
    "$$ {d_{Jaccard}}(X_i, X_j) = 1 −\\frac{|X_i ∩ X_j |}{|X_i ∪ X_j|} $$\n",
    "\n",
    "The distance simply is the number of items that Xi and Xj have in common divided by the number of unique items in both sets. For a set of m rules we can calculate the **m(m − 1)/2** distances and use them as the input for clustering. However, using clustering on the itemsets directly has several problems. First of all, data sets typically mined for association rules are high-dimensional, i.e., contain many different items. This high dimensionality is carried over to the mined rules and leads to \"curse of dimensionality\" where, due to the exponentially increasing volume, distance functions lose their usefulness. The situation is getting worse since minimum support used in association rule mining leads in addition to relatively short rules resulting in extremely sparse data. This problem is addressed in this visualization. For more reading [click here and go to page 13](https://cran.r-project.org/web/packages/arulesViz/vignettes/arulesViz.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plot(rules, method=\"grouped\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the grouped matrix plot, antecedent groups are columns and consequents are rows. The color of the balloons represent the aggregated interest measure in the group with a certain consequent and the size of the balloon shows the aggregated support. The default aggregation function is the median value in the group. The number of antecedents and the most important (frequent) items in the group are displayed as the labels for the columns. Furthermore, the columns and rows in the plot are reordered such that the aggregated interest measure is decreasing from top down and from left to right, placing the most interesting group in the top left corner. \n",
    "\n",
    "\n",
    "In above visualization the group of most interesting rules according to lift which is the default measure are shown in the top-left corner of the plot. There are 3 rules which contain “Instant food products” and up to 2 other items in the antecedent and\n",
    "the consequent is \"hamburger meat\".\n",
    "\n",
    "To increase the number of groups change k which defaults to 20."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Graph-based visualizations\n",
    "\n",
    "Visualize association rules using vertices and edges where vertices typically represent items or itemsets and edges indicate relationship in rules. Interest measures are typically added to the plot as labels on the edges or by color or width of the arrows displaying the edges. Graph-based visualization offers a very clear representation of rules but they tend to easily\n",
    "become cluttered and thus are only viable for very small sets of rules. For the following plots we select the 10 rules with the highest lift."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "subrules2 <- head(sort(rules, by=\"lift\"), 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plot(subrules2, method=\"graph\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plot(subrules2, method=\"graph\", control=list(type=\"itemsets\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above representation focuses on how the rules are composed of individual items and shows which rules share items.\n",
    "\n",
    "To explore large sets of rules with graphs, advanced interactive features like zooming, filtering, grouping and coloring nodes are needed. Such features are available in interactive visualization and exploration platforms for networks and graphs like Gephi. From arulesViz graphs for sets of association rules can be exported in the GraphML format or as a Graphviz dot-file to be explored in tools like [Gephi](https://gephi.org/). For example the 1000 rules with the highest lift are exported as.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "saveAsGraph(head(sort(rules, by=\"lift\"),1000), file=\"rules.graphml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "library(visNetwork)\n",
    "library(igraph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# ig <- plot( subrules2, method=\"graph\", control=list(type=\"items\") )\n",
    "\n",
    "# # saveAsGraph seems to render bad DOT for this case\n",
    "# tf <- tempfile( )\n",
    "# saveAsGraph( subrules2, file = tf, format = \"dot\" )\n",
    "# # clean up temp file if desired\n",
    "# #unlink(tf)\n",
    "\n",
    "# # let's bypass saveAsGraph and just use our igraph\n",
    "# ig_df <- get.data.frame( ig, what = \"both\" )\n",
    "# visNetwork(nodes = data.frame(id = ig_df$vertices$name ,value = ig_df$vertices$support ,title = ifelse(ig_df$vertices$label == \"\",ig_df$vertices$name, ig_df$vertices$label), ig_df$vertices), edges = ig_df$edges)%>%  visOptions( highlightNearest = T )\n",
    "#  #%>%  visEdges(.,style = \"arrow\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For exploring more different visuaizations [go to the page](https://cran.r-project.org/web/packages/arulesViz/vignettes/arulesViz.pdf) "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.3.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}