# Tokenizer

This class takes a set of highlighting rules, and creates a tokenizer out of them. For more information, see the wiki on extending highlighters.

### Constructors

- - new Tokenizer(Object rules, String flag)

Constructs a new tokenizer based on the given rules and flags.

Constructs a new tokenizer based on the given rules and flags.

#### Arguments

rules | Object | Required. The highlighting rules
flag | String | Required. Any additional regular expression flags to pass (like "i" for case insensitive)

### Methods

- - Tokenizer.getLineTokens(Object line, Object startState)
  - - Object

Returns an object containing two properties: `tokens`, which contains all the tokens; and `state`, the current state.

Returns an object containing two properties: `tokens`, which contains all the tokens; and `state`, the current state.

#### Arguments

line | Object | Required.
startState | Object | Required.