{"id":562,"date":"2010-10-06T22:21:06","date_gmt":"2010-10-07T03:21:06","guid":{"rendered":"http:\/\/yuguangzhang.com\/blog\/?p=562"},"modified":"2010-10-06T22:21:06","modified_gmt":"2010-10-07T03:21:06","slug":"tokenizing-prerequisites","status":"publish","type":"post","link":"http:\/\/yuguangzhang.com\/blog\/tokenizing-prerequisites\/","title":{"rendered":"Tokenizing Prerequisites"},"content":{"rendered":"

I’m taking a bite by bite approach to interpreting the course prerequisites from the scraper<\/a> to make its storage in the database possible. Meanwhile, I already finished the part of it that retrieves the data recursively<\/a>. The approach I decide to take is the same as building a compiler.<\/p>\n

I previously favored a quick approach such as the interpreter pattern.<\/p>\n

\"design\"<\/a><\/p>\n

However, commas in sentences have different meanings depending on the context.<\/p>\n

(BIOL 140, BIOL 208 or 330) and (BIOL 308 or 330)<\/pre>\n
(CS 240 or SE 240), CS 246, ECE 222<\/pre>\n
The interpreter pattern assumes a symbol only has one meaning. Thanks again to a course I took, I was already familiar with the problem and the solution. Recognizing and defining the problem was the hard part. A quick overview of the steps involved from start to finish:<\/p>\n
\"\"<\/a><\/p>\n
First step in the compiling process is to tokenize input. \u00a0That takes care of blobs such as “Level at least 3A; Not open to General Mathematics students.”. Totally useless in terms of prerequisites. So it should not show up in the token string.<\/p>\n
Building the lexer was as simple as specifying the token list, writing regular expressions for them, and setting ignored characters.
\n[cc lang=”python”]
\nimport ply.lex as lex
\nt_DEPT = r'[A-Z]{2,5}’
\nt_NUM = r’\\d{3}’
\nt_OR = r’or’
\nt_COMMA = r’,’
\nt_SEMI = r’;’
\ntokens = (
\n\t‘DEPT’,
\n\t‘NUM’,
\n\t‘OR’,
\n\t‘COMMA’,
\n\t‘SEMI’,
\n\t‘AND’,
\n\t‘LPARENS’,
\n\t‘RPARENS’,
\n)
\nt_AND = r’and’
\nt_LPARENS = r’\\(‘
\nt_RPARENS = r’\\)’
\nt_ignore  = ‘ \\t’
\ndef t_error(t):
\n\t    print “Illegal character ‘%s'” % t.value[0]
\n\t    t.lexer.skip(1)
\nlexer = lex.lex()
\ndata = ‘ (CS 240 or SE 240), CS 246, ECE 222’
\nlexer.input(data)
\nwhile True:
\n    tok = lexer.token()
\n    if not tok: break      # No more input
\n    print tok
\n[\/cc]
\nRunning it gives the output:<\/p>\n
LexToken(LPARENS,'(',1,1)\nLexToken(DEPT,'CS',1,2)\nLexToken(NUM,'240',1,5)\nLexToken(OR,'or',1,9)\nLexToken(DEPT,'SE',1,12)\nLexToken(NUM,'240',1,15)\nLexToken(RPARENS,')',1,18)\nLexToken(COMMA,',',1,19)\nLexToken(DEPT,'CS',1,21)\nLexToken(NUM,'246',1,24)\nLexToken(COMMA,',',1,27)\nLexToken(DEPT,'ECE',1,29)\nLexToken(NUM,'222',1,33)<\/pre>\n","protected":false},"excerpt":{"rendered":"
I’m taking a bite by bite approach to interpreting the course prerequisites from the scraper to make its storage in the database possible. Meanwhile, I already finished the part of it that retrieves the data recursively. The approach I decide to take is the same as building a compiler. I previously favored a quick approach […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[29],"tags":[53,22],"class_list":["post-562","post","type-post","status-publish","format-standard","hentry","category-coursetree","tag-ply","tag-python"],"aioseo_notices":[],"_links":{"self":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts\/562","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/comments?post=562"}],"version-history":[{"count":0,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts\/562\/revisions"}],"wp:attachment":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/media?parent=562"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/categories?post=562"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/tags?post=562"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}