Now breaks input lines into tokens.