Tokenizer

• Takes strings from filehandle, identifies "tokens"
• Smallest meaningful chunk
• if ( $num == 5 ) { print "bingo" }
• Written in C (toke.c)
• Deeply incestuous with parser