Open Data Search

This project is a part of the Data Science Working Group at Code for San Francisco. Other DSWG projects can be found at the main GitHub repo.

-- Project Status: Active

Project Intro/Objective

The purpose of this project is to use analytics and topic modelling of search text to improve the user experience at https://data.sfgov.org/

Partner

• SF Open Data
• https://datasf.org/opendata/
• Partner contact: Jason Lally, @jasonlally

Methods Used

• Data Analysis
• Descriptive Statistics/Data Visualization
• Natural Language Processing
• Word2Vec Modelling

Technologies

• R
• Python
  • Pandas, Spacy

Project Description

Th major goals of the project are as follows:

• Clean and process search terms and categorize search terms by quality
• Utilize Natural Language Processing and Topic Modelling on valid search terms and cluster terms to determine potential demand for data sources
• Provide actionable insights to improve search functionality on the site

Needs of this project

• NLP/Topic Modelling Expertise

Getting Started

1. Clone this repo, for help see this tutorial.

2. Data is being kept here

3. Data processing/transformation script is Data Combiner

   • Script that combines raw data from .tsv files into a single .csv file
   • Search Data Processing Jupyter Notebook - Notebook that cleans, processes and categorizes search terms

Featured Notebooks/Analysis

• Exploration of Search Google Analytics

• Search Data Modelling Juptyer Notebook

  • Notebook with vectorization of search terms using pre-trained word2vec model

Contributing DSWG Members

Team Lead (Contact): Rocio S Ng (@Rocio)

Other Members:

Name	Slack Handle
Bao Lin Liu	@jbaolinliu
Scott Brenstuhl	@scott_brenstuhl

Contact

• If you haven't joined the SF Brigade Slack, you can do that here.
• Our slack channel is #datasci-open-data_src
• Feel free to contact team leads with any questions or if you are interested in contributing!