Reads a string into the tokens the number and currency amount grammars parse.
The lexer makes one pass over the string and classifies each
character with the Localize.Number.Lexicon.t/0 it is given:
digits of the lexicon's number system and Latin digits read as
digits, the characters of the minus and plus families as signs,
the separator families as marks, and so on. A character the
lexicon does not know is classed by its Unicode general category,
from a table mix localize.gen_unicode_categories generates from
the unicode library. It drops nothing and
decides nothing: a format character, an underscore and a space are
each a token of their own, and whether a mark is a decimal separator
depends on the reading, which is the grammar's business. Every
token keeps the text it was read from, so an error can quote the
input as it was written.
Consecutive digits, consecutive digits of another number system and consecutive text characters each merge into one token; every other token is one character.
Summary
Types
The kinds of token.
A token: its kind, its value and the text it was read from. The
value differs from the text only for :digits, whose value is the
digits as Latin digits.
Functions
Returns the tokens of a string.
Types
@type kind() ::
:digits
| :other_digits
| :mark
| :space
| :minus
| :plus
| :dash
| :open
| :close
| :underscore
| :tab
| :bidi
| :format
| :text
The kinds of token.
:digits— a run of digits, with their Latin value.:other_digits— a run of digits of a number system the lexicon does not read.:mark— a punctuation mark, a final quotation mark, or a character that is a separator in some reading.:space— a space character ([:Zs:]).:minusand:plus— a sign.:dash— a dash that may stand for a zero fraction and is not a minus.:openand:close— a parenthesis.:underscore— Elixir's digit separator.:tab— a tab, which may decorate a currency amount but is no space inside a number.:bidi— a Unicode bidi control character.:format— any other format character ([:Cf:]).:text— a run of anything else.
A token: its kind, its value and the text it was read from. The
value differs from the text only for :digits, whose value is the
digits as Latin digits.
Functions
@spec tokens(String.t(), Localize.Number.Lexicon.t()) :: [token()]
Returns the tokens of a string.
Arguments
stringis the string to read.lexiconis theLocalize.Number.Lexicon.t/0to read it with.
Returns
- A list of
token/0in the order read.
Examples
iex> {:ok, en} = Localize.validate_locale(:en)
iex> {:ok, lexicon} = Localize.Number.Lexicon.fetch(en, :latn, [])
iex> Localize.Number.Lexer.tokens("USD -1,234.5", lexicon)
[
{:text, "USD", "USD"},
{:space, " ", " "},
{:minus, "-", "-"},
{:digits, "1", "1"},
{:mark, ",", ","},
{:digits, "234", "234"},
{:mark, ".", "."},
{:digits, "5", "5"}
]
iex> {:ok, ar_eg} = Localize.validate_locale(:"ar-EG")
iex> {:ok, lexicon} = Localize.Number.Lexicon.fetch(ar_eg, :arab, [])
iex> Localize.Number.Lexer.tokens("١٢٫٥", lexicon)
[{:digits, "12", "١٢"}, {:mark, "٫", "٫"}, {:digits, "5", "٥"}]