Digital Matters Workshop: Introduction to Web Scraping

Interested in automatically pulling data from a web site instead of manually cutting and pasting?  Are you a historian interested in census data?  Are you a communication scholar interested in pulling key words from presidential speeches and inserting them into a structured database?  Learn how to build a web scraper from scratch in this workshop presented by Digital Matters.

Required: a laptop with an Internet connection, and administrative privileges (in other words, you’re able to install programs yourself).

Software

Atom.io (programming text editor)

Python (programming language)

Code

Python Scripts for Workshop (Box)

Resources

Wikipedia Entry, University of Utah

National Art Gallery (archived)

Instructions for installing Python on Windows

Common Commands for Terminal (Mac OS) and Command Line (Windows/PC)

Introduction to Python

• BeautifulSoup4 Parameters
• CSV Writer Parameters