{"id":243,"date":"2010-03-15T01:57:21","date_gmt":"2010-03-15T01:57:21","guid":{"rendered":"http:\/\/yuguangzhang.com\/blog\/?p=243"},"modified":"2010-03-15T01:57:21","modified_gmt":"2010-03-15T01:57:21","slug":"uw-course-calendar-scraper","status":"publish","type":"post","link":"http:\/\/yuguangzhang.com\/blog\/uw-course-calendar-scraper\/","title":{"rendered":"UW Course Calendar Scraper"},"content":{"rendered":"

I’ve had the idea of making a self-updating, navigable tree of Waterloo courses. This is the first step. (Actually, not the first step for me. It started with Django, which had to do with my last work report’s comparison to Zen Cart. Some credit goes to Thomas Dimson for inspiration. He made the Course Qualifier<\/a>.) The main idea for this step is to gather all the information to be stored in a database. With that (the idea and plan) begins the coding phase:
\n[cc lang=”python”]
\nfrom scrapy.item import Item, Field<\/p>\n

class UcalendarItem(Item):
\n course = Field()
\n name = Field()
\n desc = Field()
\n prereq = Field()
\n offered = Field()
\n[\/cc]
\nI wanted to gather the course (“SE 101”), name (“Introduction to Methods of Software Engineering”), desc (“An introduction …”), prereq (“Software Engineering students only”), and offered (“F”) each as separate fields
\n\"snapshot4\"<\/a>
\nIn order to do that, I wrote a spider to crawl the page:
\n[cc lang=”python”]
\nfrom scrapy.spider import BaseSpider
\nfrom scrapy.selector import HtmlXPathSelector
\nfrom ucalendar.items import UcalendarItem<\/p>\n

class UcalendarSpider(BaseSpider):
\n domain_name = “uwaterloo.ca”
\n start_urls = [
\n “http:\/\/www.ucalendar.uwaterloo.ca\/0910\/COURSE\/course-SE.html”
\n ]<\/p>\n

def parse(self, response):
\n\thxs = HtmlXPathSelector(response)
\n\ttables = hxs.select(‘\/\/table[@width=”80%”]’)
\n\titems = []
\n for table in tables:
\n\t item = UcalendarItem()
\n\t item[‘desc’] = table.select(‘tr[3]\/td\/text()’).extract()
\n\t item[‘name’] = table.select(‘tr[2]\/td\/b\/text()’).extract()
\n\t item[‘course’] = table.select(‘tr[1]\/td\/b\/text()’).re(‘([A-Z]{2,5} \\d{3})’)
\n\t item[‘offered’] = table.select(‘tr[3]\/td’).re(‘.*\\[.*Offered: (F|W|S)+,* *(F|W|S)*,* *(F|W|S)*\\]’)
\n\t item[‘prereq’] = table.select(‘tr[5]\/td\/i\/text()’).re(‘([A-Z]{2,5} \\d{3})’)
\n items.append(item)
\n\treturn items<\/p>\n

SPIDER = UcalendarSpider()
\n[\/cc]
\nThere are several things to note:<\/p>\n