Parser methods, operators and combinators¶

Parser methods¶

Parser objects are returned by any of the built-in parser Parsing primitives. They can be used and manipulated as below.

class parsy.Parser¶

__init__(wrapped_fn)¶: This is a low level function to create new parsers that is used internally but is rarely needed by users of the parsy library. It should be passed a parsing function, which takes two arguments - a string/list to be parsed and the current index into the list - and returns a Result object, as described in Creating new Parser instances.

The following methods are for actually using the parsers that you have created:

parse(string_or_list)¶

Attempts to parse the given string (or list). If the parse is successful and consumes the entire string, the result is returned - otherwise, a ParseError is raised.

Instead of passing a string, you can in fact pass a list of tokens. Almost all the examples assume strings for simplicity. Some of the primitives are also clearly string specific, and a few of the combinators (such as Parser.concat()) are string specific, but most of the rest of the library will work with tokens just as well. See Separate lexing/tokenization phases for more information.

parse_partial(string_or_list)¶: Similar to parse, except that it does not require the entire string (or list) to be consumed. Returns a tuple of (result, remainder), where remainder is the part of the string (or list) that was left over.

The following methods are essentially combinators that produce new parsers from the existing one. They are provided as methods on Parser for convenience. More combinators are documented below.

desc(string)¶

Adds a desciption to the parser, which is used in the error message if parsing fails.

>>> year = regex(r'[0-9]{4}').desc('4 digit year')
>>> year.parse('123')
ParseError: expected 4 digit year at 0:0

then(other_parser)¶

Returns a parser which, if the initial parser succeeds, will continue parsing with other_parser. This will produce the value produced by other_parser.

>>> string('x').then(string('y')).parse('xy')
'y'

See also << operator.

many()¶

Returns a parser that expects the initial parser 0 or more times, and produces a list of the results. Note that this parser does not fail if nothing matches, but instead consumes nothing and produces an empty list.

>>> parser = regex(r'[a-z]').many()
>>> parser.parse('')
[]
>>> parser.parse('abc')
['a', 'b', 'c']

times(min[, max=min])¶: Returns a parser that expects the initial parser at least min times, and at most max times, and produces a list of the results. If only one argument is given, the parser is expected exactly that number of times.

at_most(n)¶: Returns a parser that expects the initial parser at most n times, and produces a list of the results.

at_least(n)¶: Returns a parser that expects the initial parser at least n times, and produces a list of the results.

optional()¶

Returns a parser that expects the initial parser zero or once, and maps the result to None in the case of no match.

>>> string('A').optional().parse('A')
'A'
>>> string('A').optional().parse('')
None

map(fn)¶

Returns a parser that transforms the produced value of the initial parser with fn.

>>> regex(r'[0-9]+').map(int).parse('1234')
1234

This is the simplest way to convert parsed strings into the data types that you need. See also combine() and combine_dict() below.

combine(fn)¶

Returns a parser that transforms the produced values of the initial parser with fn, passing the arguments using *args syntax.

Where the current parser produces an iterable of values, this can be a more convenient way to combine them than map().

Example 1 - the argument order of our callable already matches:

>>> from datetime import date
>>> yyyymmdd = seq(regex(r'[0-9]{4}').map(int),
...                regex(r'[0-9]{2}').map(int),
...                regex(r'[0-9]{2}').map(int)).combine(date)
>>> yyyymmdd.parse('20140506')
datetime.date(2014, 5, 6)

Example 2 - the argument order of our callable doesn’t match, and we need to adjust a parameter, so we can fix it using a lambda.

>>> ddmmyy = regex(r'[0-9]{2}').map(int).times(3).combine(
...                lambda d, m, y: date(2000 + y, m, d))
>>> ddmmyy.parse('060514')
datetime.date(2014, 5, 6)

The equivalent lambda to use with map would be lambda res: date(2000 + res[2], res[1], res[0]), which is less readable. The version with combine also ensures that exactly 3 items are generated by the previous parser, otherwise you get a TypeError.

combine_dict(fn)¶

Returns a parser that transforms the value produced by the initial parser (which must be a mapping/dictionary) using the supplied function/callable, passing the arguments using the **kwargs syntax.

Where the initial parser produces a dictionary of values, this can be a more convenient way to combine them than map().

This is similar to combine(), but used for with a callable that accepts keyword arguments. Compared to using combine(), by using keyword arguments instead of positional arguments we can avoid a dependence on the order of components in the string being parsed (and in the argument order of callables being used) and therefore improve flexibility, and readability.

Example using Python 3.5 and below, using tag() and map() to produce the dictionary:

>>> from datetime import date
>>> ddmmyyyy = seq(
...     regex(r'[0-9]{2}').map(int).tag('day'),
...     regex(r'[0-9]{2}').map(int).tag('month'),
...     regex(r'[0-9]{4}').map(int).tag('year'),
... ).map(dict).combine_dict(date)
>>> ddmmyyyy.parse('04052003')
datetime.date(2003, 5, 4)

With Python 3.6, we can make use of the **kwargs version of seq() to make this more readable:

>>> ddmmyyyy = seq(
...     day=regex(r'[0-9]{2}').map(int),
...     month=regex(r'[0-9]{2}').map(int),
...     year=regex(r'[0-9]{4}').map(int),
... ).combine_dict(date)
>>> ddmmyyyy.parse('04052003')
datetime.date(2003, 5, 4)

Here we used datetime.date which accepts keyword arguments. For your own parsing needs you will often use custom data types. You can create these however you like, but we recommend attrs. You can also use namedtuple from the standard library for simple cases.

tag(name)¶

Returns a parser that wraps the produced value of the initial parser in a 2 tuple containing (name, value). This is provides a very simple way to label parsed components. e.g.:

>>> day = regex(r'[0-9]+').map(int).tag('day')
>>> month = string_from("January", "February", "March", "April", "May",
...                     "June", "July", "August", "September", "October",
...                     "November", "December").tag('month')
>>> day.parse("10")
("day", 10)

>>> seq(day << whitespace, month).parse("10 September")
[('day', 10), ('month', 'September')]

It also works well when combined with .map(dict) to get a dictionary of values:

>>> seq(day << whitespace, month).map(dict).parse("10 September")
{'day': 10, 'month': 'September'}

See also combine_dict() for building objects from these dictionaries.

concat()¶

Returns a parser that concatenates together (as a string) the previously produced values. Usually used after many() and similar methods that produce multiple values.

>>> letter.at_least(1).parse("hello")
['h', 'e', 'l', 'l', 'o']
>>> letter.at_least(1).concat().parse("hello")
'hello'

result(val)¶

Returns a parser that, if the initial parser succeeds, always produces val.

>>> string('foo').result(42).parse('foo')
42

should_fail(description)¶

Returns a parser that fails when the initial parser succeeds, and succeeds when the initial parser fails (consuming no input). A description must be passed which is used in parse failure messages.

This is essentially a negative lookahead:

>>> p = letter << string(" ").should_fail("not space")
>>> p.parse('A')
'A'
>>> p.parse('A ')
ParseError: expected 'not space' at 0:1

It is also useful for implementing things like parsing repeatedly until a marker:

>>> (string(";").should_fail("not ;") >> letter).many().concat().parse_partial('ABC;')
('ABC', ';')

bind(fn)¶: Returns a parser which, if the initial parser is successful, passes the result to fn, and continues with the parser returned from fn. This is the monadic binding operation. However, since we don’t have Haskell’s do notation in Python, using this is very awkward. Instead, you should look at Generating a parser which provides a much nicer syntax for that cases where you would have needed do notation in Parsec.

sep_by(sep, min=0, max=inf)¶

Like Parser.times(), this returns a new parser that repeats the initial parser and collects the results in a list, but in this case separated by the parser sep (whose return value is discarded). By default it repeats with no limit, but minimum and maximum values can be supplied.

>>> csv = letter.at_least(1).concat().sep_by(string(","))
>>> csv.parse("abc,def")
['abc', 'def']

mark()¶

Returns a parser that wraps the initial parser’s result in a value containing column and line information of the match, as well as the original value. The new value is a 3-tuple:

((start_row, start_column),
 original_value,
 (end_row, end_column))

This is useful for being able to report problems with parsing more accurately, especially if you are using parsy as a lexer and want subsequent parsing of the token stream to be able to report original positions in error messages etc.

Parser operators¶

This section describes operators that you can use on Parser objects to build new parsers.

`|` operator¶

parser | other_parser

Returns a parser that tries parser and, if it fails, backtracks and tries other_parser. These can be chained together.

The resulting parser will produce the value produced by the first successful parser.

>>> parser = string('x') | string('y') | string('z')
>>> parser.parse('x')
'x'
>>> parser.parse('y')
'y'
>>> parser.parse('z')
'z'

>>> (string('x') >> string('y')).parse('xy')
'y'

`<<` operator¶

parser << other_parser

The same as parser.skip(other_parser) - see Parser.skip().

(Hint - the arrows point at the important parser!)

>>> (string('x') << string('y')).parse('xy')
'x'

`>>` operator¶

parser >> other_parser

The same as parser.then(other_parser) - see Parser.then().

(Hint - the arrows point at the important parser!)

>>> (string('x') >> string('y')).parse('xy')
'y'

`+` operator¶

parser1 + parser2

Requires both parsers to match in order, and adds the two results together using the + operator. This will only work if the results support the plus operator (e.g. strings and lists):

>>> (string("x") + regex("[0-9]")).parse("x1")
"x1"

>>> (string("x").many() + regex("[0-9]").map(int).many()).parse("xx123")
['x', 'x', 1, 2, 3]

The plus operator is a convenient shortcut for:

>>> seq(parser1, parser2).combine(lambda a, b: a + b)

`*` operator¶

parser1 * number

This is a shortcut for doing Parser.times():

>>> (string("x") * 3).parse("xxx")
["x", "x", "x"]

You can also set both upper and lower bounds by multiplying by a range:

>>> (string("x") * range(0, 3)).parse("xxx")
ParseError: expected EOF at 0:2

(Note the normal semantics of range are respected - the second number is an exclusive upper bound, not inclusive).

Parser combinators¶

parsy.alt(*parsers)¶

Creates a parser from the passed in argument list of alternative parsers, which are tried in order, moving to the next one if the current one fails, as per the | operator - in other words, it matches any one of the alternative parsers.

Example using *arg syntax to pass a list of parsers that have been generated by mapping string() over a list of characters:

>>> hexdigit = alt(*map(string, "0123456789abcdef"))

(In this case you would be better off using char_from())

parsy.seq(*parsers, **kw_parsers)¶

Creates a parser that runs a sequence of parsers in order and combines their results in a list.

>>> x_bottles_of_y_on_the_z = \
...    seq(regex(r"[0-9]+").map(int) << string(" bottles of "),
...        regex(r"\S+") << string(" on the "),
...        regex(r"\S+")
...        )
>>> x_bottles_of_y_on_the_z.parse("99 bottles of beer on the wall")
[99, 'beer', 'wall']

In Python 3.6, you can also use seq with keyword arguments instead of positional arguments. In this case, the produced value is a dictionary of the individual values, rather than a sequence. This can make the produced value easier to consume.

>>> name = seq(first_name=regex("\S+") << whitespace,
...            last_name=regex("\S+")
>>> name.parse("Jane Smith")
{'first_name': 'Jane',
 'last_name': 'Smith'}

Changed in version 1.1: Added **kwargs option.

Note

The **kwargs feature is for Python 3.6 and later only, because keyword arguments do not keep their order in earlier versions.

For earlier versions, see Parser.tag() for a way of labelling parsed components and producing dictionaries.

Other combinators¶

Parsy does not try to include every possible combinator - there is no reason why you cannot create your own for your needs using the built-in combinators and primitives. If you find something that is very generic and would be very useful to have as a built-in, please submit: as a PR!

Parser methods, operators and combinators¶

Parser methods¶

Parser operators¶

| operator¶

<< operator¶

>> operator¶

+ operator¶

* operator¶