# `StylesheetParser.Syntax.Tokenizer`
[🔗](https://github.com/zestcreative/stylesheet_parser/blob/v0.1.0/lib/stylesheet_parser/syntax/tokenizer.ex#L4)

The [tokenizer](https://www.w3.org/TR/css-syntax-3/#tokenization) of CSS Syntax Level 3, over
preprocessed input (`StylesheetParser.tokenize/1` turns `\r\n`, `\r` and `\f` into `\n` and NUL into
U+FFFD).

Each token is `{type, value, [line: line, column: column]}`, where it starts, columns counting
bytes from 1. Its value is its source text: a string keeps its quotes, and escapes stay escaped
(`StylesheetParser.unescape/1` decodes them). Comments are dropped, an unclosed one running to
the end.

| Type | Value | Such as |
| --- | --- | --- |
| `:ident` | the name | `color`, `--gap`, `\31 0` |
| `:function` | the name and its `(`; a `StylesheetParser.Function` among component values | `var(` |
| `:at_keyword` | the name and its `@` | `@media` |
| `:hash` | the name and its `#` | `#main`, `#f00` |
| `:string` | the string, with its quotes | `"a.png"` |
| `:bad_string` | a string with a newline in it, up to the newline | `"a` |
| `:url` | the whole `url()` without quotes, or with escapes in `url` | `url(a.png)`, `u\72l(a.png)` |
| `:bad_url` | a `url()` with whitespace, a quote or a `(` inside | `url(a b)` |
| `:number` | the number | `12`, `-0.5`, `1e3` |
| `:percentage` | the number, without its `%` | `50` for `50%` |
| `:dimension` | `[number, unit]` | `["1", "px"]` |
| `:whitespace` | the whitespace | `" "`, `"\n  "` |
| `:delim` | any other code point | `>`, `.`, `!` |
| `:colon`, `:semicolon`, `:comma` | `:`, `;`, `,` | |
| `:lparen`, `:rparen`, `:lbracket`, `:rbracket`, `:lcurly`, `:rcurly` | the bracket; a `StylesheetParser.SimpleBlock` among component values | `(` |
| `:cdo`, `:cdc` | `<!--`, `-->` | |
| `:urange` | a unicode range, only in a `unicode-range` declaration's value | `U+0-7F` |

    iex> StylesheetParser.tokenize("a { margin: 1px }")
    {:ok,
     [
       {:ident, "a", [line: 1, column: 1]},
       {:whitespace, " ", [line: 1, column: 2]},
       {:lcurly, "{", [line: 1, column: 3]},
       {:whitespace, " ", [line: 1, column: 4]},
       {:ident, "margin", [line: 1, column: 5]},
       {:colon, ":", [line: 1, column: 11]},
       {:whitespace, " ", [line: 1, column: 12]},
       {:dimension, ["1", "px"], [line: 1, column: 13]},
       {:whitespace, " ", [line: 1, column: 16]},
       {:rcurly, "}", [line: 1, column: 17]}
     ]}

# `consume`

```elixir
@spec consume(binary(), keyword()) ::
  {:ok, [term()], rest, context, line, byte_offset}
  | {:error, reason, rest, context, line, byte_offset}
when line: {pos_integer(), byte_offset},
     byte_offset: non_neg_integer(),
     rest: binary(),
     reason: String.t(),
     context: map()
```

Parses the given `binary` as consume.

Returns `{:ok, [token], rest, context, position, byte_offset}` or
`{:error, reason, rest, context, line, byte_offset}` where `position`
describes the location of the consume (start position) as `{line, offset_to_start_of_line}`.

To column where the error occurred can be inferred from `byte_offset - offset_to_start_of_line`.

## Options

  * `:byte_offset` - the byte offset for the whole binary, defaults to 0
  * `:line` - the line and the byte offset into that line, defaults to `{1, byte_offset}`
  * `:context` - the initial context value. It will be converted to a map

---

*Consult [api-reference.md](api-reference.md) for complete listing*
