# `Localize.Number.Lexer`
[🔗](https://github.com/elixir-localize/localize/blob/v1.5.0/lib/localize/number/lexer.ex#L1)

Reads a string into the tokens the number and currency amount
grammars parse.

The lexer makes one pass over the string and classifies each
character with the `t:Localize.Number.Lexicon.t/0` it is given:
digits of the lexicon's number system and Latin digits read as
digits, the characters of the minus and plus families as signs,
the separator families as marks, and so on. A character the
lexicon does not know is classed by its Unicode general category,
from a table `mix localize.gen_unicode_categories` generates from
the `unicode` library. It drops nothing and
decides nothing: a format character, an underscore and a space are
each a token of their own, and whether a mark is a decimal separator
depends on the reading, which is the grammar's business. Every
token keeps the text it was read from, so an error can quote the
input as it was written.

Consecutive digits, consecutive digits of another number system
and consecutive text characters each merge into one token; every
other token is one character.

# `kind`

```elixir
@type kind() ::
  :digits
  | :other_digits
  | :mark
  | :space
  | :minus
  | :plus
  | :dash
  | :open
  | :close
  | :underscore
  | :tab
  | :bidi
  | :format
  | :text
```

The kinds of token.

* `:digits` — a run of digits, with their Latin value.

* `:other_digits` — a run of digits of a number system the lexicon
  does not read.

* `:mark` — a punctuation mark, a final quotation mark, or a
  character that is a separator in some reading.

* `:space` — a space character (`[:Zs:]`).

* `:minus` and `:plus` — a sign.

* `:dash` — a dash that may stand for a zero fraction and is not a
  minus.

* `:open` and `:close` — a parenthesis.

* `:underscore` — Elixir's digit separator.

* `:tab` — a tab, which may decorate a currency amount but is no
  space inside a number.

* `:bidi` — a Unicode bidi control character.

* `:format` — any other format character (`[:Cf:]`).

* `:text` — a run of anything else.

# `token`

```elixir
@type token() :: {kind(), String.t(), String.t()}
```

A token: its kind, its value and the text it was read from. The
value differs from the text only for `:digits`, whose value is the
digits as Latin digits.

# `tokens`

```elixir
@spec tokens(String.t(), Localize.Number.Lexicon.t()) :: [token()]
```

Returns the tokens of a string.

### Arguments

* `string` is the string to read.

* `lexicon` is the `t:Localize.Number.Lexicon.t/0` to read it with.

### Returns

* A list of `t:token/0` in the order read.

### Examples

    iex> {:ok, en} = Localize.validate_locale(:en)
    iex> {:ok, lexicon} = Localize.Number.Lexicon.fetch(en, :latn, [])
    iex> Localize.Number.Lexer.tokens("USD -1,234.5", lexicon)
    [
      {:text, "USD", "USD"},
      {:space, " ", " "},
      {:minus, "-", "-"},
      {:digits, "1", "1"},
      {:mark, ",", ","},
      {:digits, "234", "234"},
      {:mark, ".", "."},
      {:digits, "5", "5"}
    ]

    iex> {:ok, ar_eg} = Localize.validate_locale(:"ar-EG")
    iex> {:ok, lexicon} = Localize.Number.Lexicon.fetch(ar_eg, :arab, [])
    iex> Localize.Number.Lexer.tokens("١٢٫٥", lexicon)
    [{:digits, "12", "١٢"}, {:mark, "٫", "٫"}, {:digits, "5", "٥"}]

---

*Consult [api-reference.md](api-reference.md) for complete listing*