early-access version 3088

This commit is contained in:
pineappleEA
2022-11-05 15:35:56 +01:00
parent 4e4fc25ce3
commit b601909c6d
35519 changed files with 5996896 additions and 860 deletions

View File

@@ -0,0 +1,65 @@
#==============================================================================
# Copyright (c) 2001-2011 Joel de Guzman
# Copyright (c) 2001-2011 Hartmut Kaiser
#
# Use, modification and distribution is subject to the Boost Software
# License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at
# http://www.boost.org/LICENSE_1_0.txt)
#==============================================================================
project spirit/doc ;
import boostbook ;
import quickbook ;
import modules ;
path-constant here : . ;
if --enable-index in [ modules.peek : ARGV ]
{
ECHO "Building the Spirit docs with automatic index generation enabled." ;
using auto-index ;
project : requirements
<quickbook-define>__use_auto_index__
<auto-index>on
<auto-index-script>$(here)/index.idx
<format>pdf:<auto-index-internal>off
<format>html:<auto-index-internal>on
<xsl:param>index.on.type=1
;
}
else
{
ECHO "Building the Spirit docs with automatic index generation disabled. Try building with --enable-index." ;
}
path-constant images_location : html ;
boostbook spirit2
:
spirit2.qbk
:
<xsl:param>boost.root=../../../..
<xsl:param>chunk.section.depth=4
<xsl:param>chunk.first.sections=1
<xsl:param>toc.section.depth=3
<xsl:param>toc.max.depth=3
<xsl:param>generate.section.toc.level=4
<format>html:<xsl:param>admon.graphics.path=images/
<include>.
<format>pdf:<xsl:param>img.src.path=$(images_location)/
<format>pdf:<xsl:param>draft.mode="no"
<format>pdf:<xsl:param>"boost.url.prefix=http://www.boost.org/doc/libs/release/libs/spirit/doc/html"
;
###############################################################################
alias boostdoc ;
explicit boostdoc ;
alias boostrelease
:
spirit2
../repository/doc//spirit2_repository
x3//spirit_x3
;
explicit boostrelease ;

View File

@@ -0,0 +1,27 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section Abstracts]
[include abstracts/syntax_diagram.qbk]
[include abstracts/peg.qbk]
[/include abstracts/parsing.qbk]
[/include abstracts/generating.qbk]
[/include abstracts/primitives.qbk]
[/include abstracts/operators.qbk]
[include abstracts/attributes.qbk]
[/include abstracts/semantic_actions.qbk]
[/include abstracts/directives.qbk]
[/include abstracts/rules.qbk]
[/include abstracts/grammars.qbk]
[/include abstracts/debugging.qbk]
[/include abstracts/error_handling.qbk]
[/include abstracts/parse_trees_and_asts.qbk]
[endsect]

View File

@@ -0,0 +1,310 @@
[/==============================================================================
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2001-2011 Joel de Guzman
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:attributes Attributes]
[/////////////////////////////////////////////////////////////////////////////]
[section:primitive_attributes Attributes of Primitive Components]
Parsers and generators in __spirit__ are fully attributed. __qi__ parsers always
/expose/ an attribute specific to their type. This is called /synthesized
attribute/ as it is returned from a successful match representing the matched
input sequence. For instance, numeric parsers, such as `int_` or `double_`,
return the `int` or `double` value converted from the matched input sequence.
Other primitive parser components have other intuitive attribute types, such as
for instance `int_` which has `int`, or `ascii::char_` which has `char`. For
primitive parsers apply the normal C++ convertibility rules: you can use any
C++ type to receive the parsed value as long as the attribute type of the
parser is convertible to the type provided. The following example shows how a
synthesized parser attribute (the `int` value) is extracted by calling the
API function `qi::parse`:
int value = 0;
std::string str("123");
std::string::iterator strbegin = str.begin();
qi::parse(strbegin, str.end(), int_, value); // value == 123
The attribute type of a generator defines what data types this generator is
able to consume in order to produce its output. __karma__ generators always
/expect/ an attribute specific to their type. This is called /consumed
attribute/ and is expected to be passed to the generator. The consumed
attribute is most of the time the value the generator is designed to emit
output for. For primitive generators the normal C++ convertibility rules apply.
Any data type convertible to the attribute type of a primitive generator can be
used to provide the data to generate. We present a similar example as above,
this time the consumed attribute of the `int_` generator (the `int` value)
is passed to the API function `karma::generate`:
int value = 123;
std::string str;
std::back_insert_iterator<std::string> out(str);
karma::generate(out, int_, value); // str == "123"
Other primitive generator components have other intuitive attribute types, very
similar to the corresponding parser components. For instance, the
`ascii::char_` generator has `char` as consumed attribute. For a full list of
available parser and generator primitives and their attribute types please see
the sections __sec_qi_primitive__ and __sec_karma_primitive__.
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:compound_attributes Attributes of Compound Components]
__qi__ and __karma__ implement well defined attribute type propagation rules
for all compound parsers and generators, such as sequences, alternatives,
Kleene star, etc. The main attribute propagation rule for a sequences is for
instance:
[table
[[Library] [Sequence attribute propagation rule]]
[[Qi] [`a: A, b: B --> (a >> b): tuple<A, B>`]]
[[Karma] [`a: A, b: B --> (a << b): tuple<A, B>`]]
]
which reads as:
[:Given `a` and `b` are parsers (generators), and `A` is the attribute type of
`a`, and `B` is the attribute type of `b`, then the attribute type of
`a >> b` (`a << b`) will be `tuple<A, B>`.]
[note The notation `tuple<A, B>` is used as a placeholder expression for any
fusion sequence holding the types A and B, such as
`boost::fusion::tuple<A, B>` or `std::pair<A, B>` (for more information
see __fusion__).]
As you can see, in order for a type to be compatible with the attribute type
of a compound expression it has to
* either be convertible to the attribute type,
* or it has to expose certain functionalities, i.e. it needs to conform to a
concept compatible with the component.
Each compound component implements its own set of attribute propagation rules.
For a full list of how the different compound generators consume attributes
see the sections __sec_qi_compound__ and __sec_karma_compound__.
[heading The Attribute of Sequence Parsers and Generators]
Sequences require an attribute type to expose the concept of a fusion sequence,
where all elements of that fusion sequence have to be compatible with the
corresponding element of the component sequence. For example, the expression:
[table
[[Library] [Sequence expression]]
[[Qi] [`double_ >> double_`]]
[[Karma] [`double_ << double_`]]
]
is compatible with any fusion sequence holding two types, where both types have
to be compatible with `double`. The first element of the fusion sequence has to
be compatible with the attribute of the first `double_`, and the second element
of the fusion sequence has to be compatible with the attribute of the second
`double_`. If we assume to have an instance of a `std::pair<double, double>`,
we can directly use the expressions above to do both, parse input to fill the
attribute:
// the following parses "1.0 2.0" into a pair of double
std::string input("1.0 2.0");
std::string::iterator strbegin = input.begin();
std::pair<double, double> p;
qi::phrase_parse(strbegin, input.end(),
qi::double_ >> qi::double_, // parser grammar
qi::space, // delimiter grammar
p); // attribute to fill while parsing
and generate output for it:
// the following generates: "1.0 2.0" from the pair filled above
std::string str;
std::back_insert_iterator<std::string> out(str);
karma::generate_delimited(out,
karma::double_ << karma::double_, // generator grammar (format description)
karma::space, // delimiter grammar
p); // data to use as the attribute
(where the `karma::space` generator is used as the delimiter, allowing to
automatically skip/insert delimiting spaces in between all primitives).
[tip *For sequences only:* __qi__ and __karma__ expose a set of API functions
usable mainly with sequences. Very much like the functions of the `scanf`
and `printf` families these functions allow to pass the attributes for
each of the elements of the sequence separately. Using the corresponding
overload of /Qi's/ parse or /Karma's/ `generate()` the expression above
could be rewritten as:
``
double d1 = 0.0, d2 = 0.0;
qi::phrase_parse(begin, end, qi::double_ >> qi::double_, qi::space, d1, d2);
karma::generate_delimited(out, karma::double_ << karma::double_, karma::space, d1, d2);
``
where the first attribute is used for the first `double_`, and
the second attribute is used for the second `double_`.
]
[heading The Attribute of Alternative Parsers and Generators]
Alternative parsers and generators are all about - well - alternatives. In
order to store possibly different result (attribute) types from the different
alternatives we use the data type __boost_variant__. The main attribute
propagation rule of these components is:
a: A, b: B --> (a | b): variant<A, B>
Alternatives have a second very important attribute propagation rule:
a: A, b: A --> (a | b): A
often allowing to simplify things significantly. If all sub expressions of
an alternative expose the same attribute type, the overall alternative
will expose exactly the same attribute type as well.
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:more_compound_attributes More About Attributes of Compound Components]
While parsing input or generating output it is often desirable to combine some
constant elements with variable parts. For instance, let us look at the example
of parsing or formatting a complex number, which is written as `(real, imag)`,
where `real` and `imag ` are the variables representing the real and imaginary
parts of our complex number. This can be achieved by writing:
[table
[[Library] [Sequence expression]]
[[Qi] [`'(' >> double_ >> ", " >> double_ >> ')'`]]
[[Karma] [`'(' << double_ << ", " << double_ << ')'`]]
]
Fortunately, literals (such as `'('` and `", "`) do /not/ expose any attribute
(well actually, they do expose the special type `unused_type`, but in this
context `unused_type` is interpreted as if the component does not expose any
attribute at all). It is very important to understand that the literals don't
consume any of the elements of a fusion sequence passed to this component
sequence. As said, they just don't expose any attribute and don't produce
(consume) any data. The following example shows this:
// the following parses "(1.0, 2.0)" into a pair of double
std::string input("(1.0, 2.0)");
std::string::iterator strbegin = input.begin();
std::pair<double, double> p;
qi::parse(strbegin, input.end(),
'(' >> qi::double_ >> ", " >> qi::double_ >> ')', // parser grammar
p); // attribute to fill while parsing
and here is the equivalent __karma__ code snippet:
// the following generates: (1.0, 2.0)
std::string str;
std::back_insert_iterator<std::string> out(str);
generate(out,
'(' << karma::double_ << ", " << karma::double_ << ')', // generator grammar (format description)
p); // data to use as the attribute
where the first element of the pair passed in as the data to generate is still
associated with the first `double_`, and the second element is associated with
the second `double_` generator.
This behavior should be familiar as it conforms to the way other input and
output formatting libraries such as `scanf`, `printf` or `boost::format` are
handling their variable parts. In this context you can think about __qi__'s
and __karma__'s primitive components (such as the `double_` above) as of being
type safe placeholders for the attribute values.
[tip Similarly to the tip provided above, this example could be rewritten
using /Spirit's/ multi-attribute API function:
``
double d1 = 0.0, d2 = 0.0;
qi::parse(begin, end, '(' >> qi::double_ >> ", " >> qi::double_ >> ')', d1, d2);
karma::generate(out, '(' << karma::double_ << ", " << karma::double_ << ')', d1, d2);
``
which provides a clear and comfortable syntax, more similar to the
placeholder based syntax as exposed by `printf` or `boost::format`.
]
Let's take a look at this from a more formal perspective. The sequence attribute
propagation rules define a special behavior if generators exposing `unused_type`
as their attribute are involved (see __sec_karma_compound__):
[table
[[Library] [Sequence attribute propagation rule]]
[[Qi] [`a: A, b: Unused --> (a >> b): A`]]
[[Karma] [`a: A, b: Unused --> (a << b): A`]]
]
which reads as:
[:Given `a` and `b` are parsers (generators), and `A` is the attribute type of
`a`, and `unused_type` is the attribute type of `b`, then the attribute type
of `a >> b` (`a << b`) will be `A` as well. This rule applies regardless of
the position the element exposing the `unused_type` is at.]
This rule is the key to the understanding of the attribute handling in
sequences as soon as literals are involved. It is as if elements with
`unused_type` attributes 'disappeared' during attribute propagation. Notably,
this is not only true for sequences but for any compound components. For
instance, for alternative components the corresponding rule is:
a: A, b: Unused --> (a | b): A
again, allowing to simplify the overall attribute type of an expression.
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:nonterminal_attributes Attributes of Rules and Grammars]
Nonterminals are well known from parsers where they are used as the main means
of constructing more complex parsers out of simpler ones. The nonterminals in
the parser world are very similar to functions in an imperative programming
language. They can be used to encapsulate parser expressions for a particular
input sequence. After being defined, the nonterminals can be used as 'normal'
parsers in more complex expressions whenever the encapsulated input needs to be
recognized. Parser nonterminals in __qi__ may accept /parameters/ (inherited
attributes) and usually return a value (the synthesized attribute).
Both, the types of the inherited and the synthesized attributes have to be
explicitly specified while defining the particular `grammar` or the `rule`
(the Spirit __repo__ additionally has `subrules` which conform to a similar
interface). As an example, the following code declares a __qi__ `rule`
exposing an `int` as its synthesized attribute, while expecting a single
`double` as its inherited attribute (see the section about the __qi__ __rule__
for more information):
qi::rule<Iterator, int(double)> r;
In the world of generators, nonterminals are just as useful as in the parser
world. Generator nonterminals encapsulate a format description for a particular
data type, and, whenever we need to emit output for this data type, the
corresponding nonterminal is invoked in a similar way as the predefined
__karma__ generator primitives. The __karma__ [karma_nonterminal nonterminals]
are very similar to the __qi__ nonterminals. Generator nonterminals may accept
/parameters/ as well, and we call those inherited attributes too. The main
difference is that they do not expose a synthesized attribute (as parsers do),
but they require a special /consumed attribute/. Usually the consumed attribute
is the value the generator creates its output from. Even if the consumed
attribute is not 'returned' from the generator we chose to use the same
function style declaration syntax as used in __qi__. The example below declares
a __karma__ `rule` consuming a `double` while not expecting any additional
inherited attributes.
karma::rule<OutputIterator, double()> r;
The inherited attributes of nonterminal parsers and generators are normally
passed to the component during its invocation. These are the /parameters/ the
parser or generator may accept and they can be used to parameterize the
component depending on the context they are invoked from.
[/
* attribute propagation
* explicit and operator%=
]
[endsect]
[endsect] [/ Attributes]

View File

@@ -0,0 +1,113 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section Parsing Expression Grammar]
Parsing Expression Grammars (PEG) [footnote Bryan Ford: Parsing Expression
Grammars: A Recognition-Based Syntactic Foundation,
[@http://pdos.csail.mit.edu/~baford/packrat/popl04/]] are a derivative of
Extended Backus-Naur Form (EBNF) [footnote Richard E. Pattis: EBNF: A Notation
to Describe Syntax, [@http://www.cs.cmu.edu/~pattis/misc/ebnf.pdf]]
with a different interpretation, designed to represent a recursive descent
parser. A PEG can be directly represented as a recursive-descent parser.
Like EBNF, PEG is a formal grammar for describing a formal language in
terms of a set of rules used to recognize strings of this language.
Unlike EBNF, PEGs have an exact interpretation. There is only one valid
parse tree (see __ast__) for each PEG grammar.
[heading Sequences]
Sequences are represented by juxtaposition like in EBNF:
a b
The PEG expression above states that, in order for this to succeed,
`b` must follow `a`. Here's the syntax diagram:
[:__sd_sequence__]
Here's a trivial example:
'x' digit
which means the character `x` must be followed by a digit.
[note In __qi__, we use the `>>` for sequences since C++ does not
allow juxtaposition, while in __karma__ we use the `<<` instead.]
[heading Alternatives]
Alternatives are represented in PEG using the slash:
a / b
[note In __qi__ and __karma__, we use the `|` for alternatives just as in EBNF.]
Alternatives allow for choices. The expression above reads: try to match
`a`. If `a` succeeds, success, if not try to match `b`. This is a bit of
a deviation from the usual EBNF interpretation where you simply match
`a` *or* `b`. Here's the syntax diagram:
[:__sd_choice__]
PEGs allow for ambiguity in the alternatives. In the expression above,
both `a` or `b` can both match an input string. However, only the first
matching alternative is valid. As noted, there can only be one valid
parse tree. [/FIXME: $$$ explain more about this $$$]
[heading Loops]
Again, like EBNF, PEG uses the regular-expression Kleene star and the
plus loops:
a*
a+
[note __qi__ and __karma__ use the prefix star and plus since there is no
postfix star or plus in C++.]
Here are the syntax diagrams:
[:__sd_kleene__]
[:__sd_plus__]
The first, called the Kleene star, matches zero or more of its subject
`a`. The second, plus, matches one ore more of its subject `a`.
Unlike EBNF, PEGs have greedy loops. It will match as much as it can
until its subject fails to match without regard to what follows. The
following is a classic example of a fairly common EBNF/regex expression
failing to match in PEG:
alnum* digit
In PEG, alnum will eat as much alpha-numeric characters as it can
leaving nothing more left behind. Thus, the trailing digit will get
nothing. Loops are simply implemented in recursive descent code as
for/while loops making them extremely efficient. That is a definite
advantage. On the other hand, those who are familiar with EBNF and regex
behavior might find the behavior a major gotcha. PEG provides a couple
of other mechanisms to circumvent this. We will see more of these other
mechanisms shortly.
[heading Difference]
In some cases, you may want to restrict a certain expression. You can
think of a PEG expression as a match for a potentially infinite set of
strings. The difference operator allows you to restrict this set:
a - b
The expression reads: match `a` but not `b`.
[note There is no difference operator in __karma__, as the concept does not
make sense in the context of output generation.]
[endsect]

View File

@@ -0,0 +1,104 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section Syntax Diagram]
In the next section, we will deal with Parsing Expression Grammars
(PEG) [footnote Bryan Ford: Parsing Expression Grammars: A Recognition-Based
Syntactic Foundation, [@http://pdos.csail.mit.edu/~baford/packrat/popl04/]],
a variant of Extended Backus-Naur Form (EBNF) [footnote Richard E. Pattis: EBNF:
A Notation to Describe Syntax, [@http://www.cs.cmu.edu/~pattis/misc/ebnf.pdf]]
with a different interpretation. It is easier to understand PEG using Syntax
Diagrams. Syntax diagrams represent a grammar graphically. It was used extensively
by Niklaus Wirth [footnote Niklaus Wirth: The Programming Language
Pascal. (July 1973)] in the "Pascal User Manual". Syntax Diagrams are
easily understandable by programmers due to their similarity to flow
charts. The isomorphism of the diagrams and functions make them ideal for
representing __rd__ parsers which are essentially mutually recursive
functions.
Historically, Parsing Expression Grammars have been used for describing grammars
for parsers only (hence the name). In __spirit__ we use a very similar notation
for output generation as well. Almost all the concepts described here are
equally applicable both to __qi__ parsers and to __karma__ generators.
[heading Elements]
All diagrams have one entry and one exit point. Arrows connect all possible
paths through the grammar from the entry point to the exit point.
[:__sd_start_stop__]
Terminals are represented by round boxes. Terminals are atomic and
usually represent plain characters, strings or tokens.
[:__sd_terminals__]
Non-terminals are represented by boxes. Diagrams are modularized using
named non-terminals. A complex diagram can be broken down into a set of
non-terminals. Non-terminals also allow recursion (i.e. a non-terminal
can call itself).
[:__sd_non_terminals__]
[heading Constructs]
The most basic composition is the Sequence. B follows A:
[:__sd_sequence__]
The ordered choice henceforth we will call /alternatives/. In PEG,
ordered choice and alternatives are not quite the same. PEG allows
ambiguity of choice where one or more branches can succeed. In PEG, in
case of ambiguity, the first one always wins.
[:__sd_choice__]
The optional (zero-or-one):
[:__sd_optional__]
Now, the loops. We have the zero-or-more and one-or-more:
[:__sd_kleene__]
[:__sd_plus__]
Take note that, as in PEG, these loops behave greedily. If there is
another 'A' just before the end-point, it will always fail because the
preceding loop has already exhausted all 'A's and there is nothing more
left. This is a crucial difference between PEG and general Context Free
Grammars (CFGs). This behavior is quite obvious with syntax diagrams as
they resemble flow-charts.
[heading Predicates]
Now, the following are Syntax Diagram versions of PEG predicates. These
are not traditionally found in Syntax Diagrams. These are special
extensions we invented to closely follow PEGs.
First, we introduce a new element, the Predicate:
[:__sd_predicate__]
This is similar to the conditionals in flow charts where the 'No' branch
is absent and always signals a failed parse.
We have two versions of the predicate, the /And-Predicate/ and the
/Not-Predicate/:
[:__sd_and_predicate__]
[:__sd_not_predicate__]
The /And-Predicate/ tries the predicate, P, and succeeds if P succeeds,
or otherwise fail. The opposite is true with the /Not-Predicate/. It
tries the predicate, P, and fails if P succeeds, or otherwise succeeds.
Both versions do a look-ahead but do not consume any input regardless if
P succeeds or not.
[endsect]

View File

@@ -0,0 +1,230 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section Acknowledgments]
This version of Spirit is a complete rewrite of the /classic/ Spirit many
people have been contributing to (see below). But there are a couple of people
who already managed to help significantly during this rewrite. We would like to
express our special acknowledgement to (in no particular order):
[*Eric Niebler] for writing Boost.Proto, without which this rewrite wouldn't
have been possible, and helping with examples, advice, and suggestions on
how to use Boost.Proto in the best possible way.
[*Ben Hanson] for providing us with an early version of his __lexertl__ library,
which is proposed to be included into Boost (as Boost.Lexer). At the time
of this writing the Boost review for this library is still pending.
[*Francois Barel] for his silent but steady work on making and keeping Spirit
compatible with all versions of gcc, older and newest ones. He not only
contributed subrules to Spirit V2.1, but always keeps an eye on the small
details which are so important to make a difference.
[*Andreas Haberstroh] for proof reading the documentation and fixing those
non-native-speaker-quirks we managed to introduce into the first versions of
the documentation.
[*Chris Hoeppler] for taking up the editorial tasks for the initial
version of this documentation together with Andreas Haberstroh. Chris
did a lot especially at the last minute when we are about to release.
[*Michael Caisse] (a.k.a mjcaisse) for being Spirit's benevolent
evangelist on freenode's #boost IRC . No Spirit question asked there
stays without an answer because of his active involvement. Also, we would
like to thank him for last minute editing work on the 2.1 release
documentation.
[*Tobias Schwinger] for proposing expectation points and GCC port of an
early version.
[*Dave Abrahams] as always, for countless advice and help on C++,
library development, interfaces, usability and ease of use, for
reviewing the code and providing valuable feedback and for always
keeping us on our toes.
[*OvermindDL] for his creative ideas on the mailing list helping to resolve
even more difficult user problems.
[*Carl Barron] for his early adoption and valuable feedback on the Lexer
library forcing us to design a proper API covering all of his use cases.
He also contributed an early version of the variadic attribute API for
Qi.
[*Daniel James] for improving and maintaining Quickbook, the tool we use
for this documentation. Also, for bits and pieces here and there such
documentation suggestions and editorial patches.
[*Stephan Menzel] for his early adoption of Qi and Karma and his willingness
to invest time to spot bugs which were hard to isolate. Also, for his feedback
on the documentation.
[*Ray Burkholder] and [*Dainis Polis] for last minute feedback on the
documentation.
[*Steve Brandt] for his his effort trying to put Qi and Karma to some use while
writing a source to source language transformation tool. He made many incredible
suggestions helping to improve the usability of both libraries.
[*Bryce Leylbach] (a.k.a. wash) for contributing the components `qi::`__qi_as__
and `karma::`__karma_as__, for working on __utree__ and its integration with
Qi and Karma, for adding numeric parser literals, for contributing the related
documentation, and for being a big help in the everyday maintenance of the
library's (and our) sanity.
[*Mathias Gaunard] for his bug reports, suggestions and regression test
contributions to the Lexer helping to find subtle bugs and to improve its
overall usability.
[*Thomas Bernard] (a.k.a. teajay) for working on the keyword parser and for
adding explicit names to the `qi::`__qi_symbols__ component.
[*Robert Stewart] for his active participation on the mailing list, his
helpful comments, examples and suggestions.
[*Thomas Taylor], [*Richard Crossley], [*Semen], and [*Adalberto Castelo] for
their help in isolating problems while testing the new attribute handling code
released with V2.5.
Special thanks to spirit-devel and spirit-general mailing lists for
participating in the discussions, being early adopters of pre-release
versions of Spirit2 from the very start and helping out in various tasks
such as helping with support, bug tracking, benchmarking and testing,
etc. The list include: [*Larry Evans], [*Richard Webb], [*Martin Wille],
[*Dan Marsden], [*Cedric Venet], [*Allan Odgaard], [*Matthias
Vallentin], [*Justinas V.D.], [*Darid Tromer], [*Brian O'Kennedy],
[*Aaron Graham], [*Joerg Becker].
[*Joao Abecasis] for his early support and involvement in Spirit2
development and for disturbing my peace every once in a while for a
couple of jokes.
The list goes on and on... if you've been mentioned thank Joel and
Hartmut, if not, kick Joao :-)
[heading Acknowledgements from the Spirit V1 /classic/ Documentation]
Special thanks for working on Spirit /classic/ to:
[*Dan Nuffer] for his work on lexers, parse trees, ASTs, XML parsers, the
multi-pass iterator as well as administering Spirit's site, editing,
maintaining the CVS and doing the releases plus a zillion of other chores that
were almost taken for granted.
[*Hartmut Kaiser] for his work on the C parser, the work on the C/C++
preprocessor, utility parsers, the original port to Intel 5.0, various work on
Phoenix, porting to v1.5, the meta-parsers, the grouping-parsers, extensive
testing and painstaking attention to details.
[*Martin Wille] who improved grammar multi thread safety, contributed the eol_p
parser, the dynamic parsers, documentation and for taking an active role in
almost every aspect from brainstorming and design to coding. And, as always,
helps keep the regression tests for g++ on Linux as green as ever :-).
[*Martijn W. Van Der Lee] our Web site administrator and for contributing the
RFC821 parser.
[*Giovanni Bajo] for last minute tweaks of Spirit 1.8.0 for CodeWarrior 8.3.
Actually, I'm ashamed Giovanni was not in this list already. He's done a lot
since Spirit 1.5, the first Boost.Spirit release. He's instrumental in the
porting of the Spirit iterators stuff to the new Boost Iterators Library
(version 2). He also did various bug fixes and wrote some tests here and there.
[*Juan Carlos Arevalo-Baeza (JCAB)*] for his work on the C++ parser, the position
iterator, ports to v1.5 and keeping the mailing list discussions alive and
kicking.
[*Vaclav Vesely], lots of stuff, the no\_actions directive, various patches
fixes, the distinct parsers, the lazy parser, some phoenix tweaks and add-ons
(e.g. new\_). Also, *Stefan Slapeta] and wife for editing Vaclav's distinct
parser doc.
[*Raghavendra Satish] for doing the original v1.3 port to VC++ and his work on
Phoenix.
[*Noah Stein] for following up and helping Ragav on the VC++ ports.
[*Hakki Dogusan], for his original v1.0 Pascal parser.
[*John (EBo) David] for his work on the VM and watching over my shoulder as I
code giving the impression of distance eXtreme programming.
[*Chris Uzdavinis] for feeding in comments and valuable suggestions as well as
editing the documentation.
[*Carsten Stoll], for his work on dynamic parsers.
[*Andy Elvey] and his conifer parser.
[*Bruce Florman], who did the original v1.0 port to VC++.
[*Jeff Westfahl] for porting the loop parsers to v1.5 and contributing the file
iterator.
[*Peter Simons] for the RFC date parser example and tutorial plus helping out
with some nitty gritty details.
[*Markus Sch'''&ouml;'''pflin] for suggesting the end_p parser and lots of other
nifty things and his active presence in the mailing list.
[*Doug Gregor] for mentoring and his ability to see things that others don't.
[*David Abrahams] for giving Joel a job that allows him to still work on Spirit,
plus countless advice and help on C++ and specifically template
metaprogramming.
[*Aleksey Gurtovoy] for his MPL library from which we stole many metaprogramming
tricks especially for less conforming compilers such as Borland and VC6/7.
[*Gustavo Guerra] for his last minute review of Spirit and constant feedback,
plus patches here and there (e.g. proposing the new dot behavior of the real
numerics parsers).
[*Nicola Musatti], [*Paul Snively], [*Alisdair Meredith] and [*Hugo Duncan] for
testing and sending in various patches.
[*Steve Rowe] for his splendid work on the TSTs that will soon be taken into
Spirit.
[*Jonathan de Halleux] for his work on actors.
[*Angus Leeming] for last minute editing work on the 1.8.0 release
documentation, his work on Phoenix and his active presence in the Spirit
mailing list.
[*Joao Abecasis] for his active presence in the Spirit mailing list, providing
user support, participating in the discussions and so on.
[*Guillaume Melquiond] for a last minute patch to multi_pass for 1.8.1.
[*Peder Holt] for his porting work on Phoenix, Fusion and Spirit to VC6.
To Joel's wife Mariel who did the graphics in this document.
My, there's a lot in this list! And it's a continuing list. We add people to
this list every time. We hope we did not forget anyone. If we missed
someone you know who has helped in any way, please inform us.
Special thanks also to people who gave feedback and valuable comments,
particularly members of Boost and Spirit mailing lists. This includes all those
who participated in the review:
[*John Maddock], our review manager, [*Aleksey Gurtovoy], [*Andre Hentz],
[*Beman Dawes], [*Carl Daniel], [*Christopher Currie], [*Dan Gohman],
[*Dan Nuffer], [*Daryle Walker], [*David Abrahams], [*David B. Held],
[*Dirk Gerrits], [*Douglas Gregor], [*Hartmut Kaiser], [*Iain K.Hanson],
[*Juan Carlos Arevalo-Baeza], [*Larry Evans], [*Martin Wille],
[*Mattias Flodin], [*Noah Stein], [*Nuno Lucas], [*Peter Dimov],
[*Peter Simons], [*Petr Kocmid], [*Ross Smith], [*Scott Kirkwood],
[*Steve Cleary], [*Thorsten Ottosen], [*Tom Wenisch], [*Vladimir Prus]
Finally thanks to SourceForge for hosting the Spirit project and Boost: a C++
community comprised of extremely talented library authors who participate in
the discussion and peer review of well crafted C++ libraries.
[endsect]

View File

@@ -0,0 +1,14 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section Advanced]
[include advanced/indepth.qbk]
[include advanced/customization_points.qbk]
[endsect]

View File

@@ -0,0 +1,280 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2009 Andreas Haberstroh?
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:indepth In Depth]
[section:parsers_indepth Parsers in Depth]
This section is not for the faint of heart. In here, are distilled the inner
workings of __qi__ parsers, using real code from the __spirit__ library as
examples. On the other hand, here is no reason to fear reading on, though.
We tried to explain things step by step while highlighting the important
insights.
The `__parser_concept__` class is the base class for all parsers.
[import ../../../../boost/spirit/home/qi/parser.hpp]
[parser_base_parser]
The `__parser_concept__` class does not really know how to parse anything but
instead relies on the template parameter `Derived` to do the actual parsing.
This technique is known as the "Curiously Recurring Template Pattern" in template
meta-programming circles. This inheritance strategy gives us the power of
polymorphism without the virtual function overhead. In essence this is a way to
implement compile time polymorphism.
The Derived parsers, `__primitive_parser_concept__`, `__unary_parser_concept__`,
`__binary_parser_concept__` and `__nary_parser_concept__` provide the necessary
facilities for parser detection, introspection, transformation and visitation.
Derived parsers must support the following:
[variablelist bool parse(f, l, context, skip, attr)
[[`f`, `l`] [first/last iterator pair]]
[[`context`] [enclosing rule context (can be unused_type)]]
[[`skip`] [skipper (can be unused_type)]]
[[`attr`] [attribute (can be unused_type)]]
]
The /parse/ is the main parser entry point. /skipper/ can be an `unused_type`.
It's a type used every where in __spirit__ to signify "don't-care". There
is an overload for /skip/ for `unused_type` that is simply a no-op.
That way, we do not have to write multiple parse functions for
phrase and character level parsing.
Here are the basic rules for parsing:
* The parser returns `true` if successful, `false` otherwise.
* If successful, `first` is incremented N number of times, where N
is the number of characters parsed. N can be zero --an empty (epsilon)
match.
* If successful, the parsed attribute is assigned to /attr/
* If unsuccessful, `first` is reset to its position before entering
the parser function. /attr/ is untouched.
[variablelist void what(context)
[[`context`] [enclosing rule context (can be `unused_type`)]]
]
The /what/ function should be obvious. It provides some information
about ["what] the parser is. It is used as a debugging aid, for
example.
[variablelist P::template attribute<context>::type
[[`P`] [a parser type]]
[[`context`] [A context type (can be unused_type)]]
]
The /attribute/ metafunction returns the expected attribute type
of the parser. In some cases, this is context dependent.
In this section, we will dissect two parser types:
[variablelist Parsers
[[`__primitive_parser_concept__`] [A parser for primitive data (e.g. integer parsing).]]
[[`__unary_parser_concept__`] [A parser that has single subject (e.g. kleene star).]]
]
[/------------------------------------------------------------------------------]
[heading Primitive Parsers]
For our dissection study, we will use a __spirit__ primitive, the `any_int_parser`
in the boost::spirit::qi namespace.
[import ../../../../boost/spirit/home/qi/numeric/int.hpp]
[primitive_parsers_any_int_parser]
The `any_int_parser` is derived from a `__primitive_parser_concept__<Derived>`,
which in turn derives from `parser<Derived>`. Therefore, it supports the
following requirements:
* The `parse` member function
* The `what` member function
* The nested `attribute` metafunction
/parse/ is the main entry point. For primitive parsers, our first thing to do is
call:
``
qi::skip(first, last, skipper);
``
to do a pre-skip. After pre-skipping, the parser proceeds to do its thing. The
actual parsing code is placed in `extract_int<T, Radix, MinDigits,
MaxDigits>::call(first, last, attr);`
This simple no-frills protocol is one of the reasons why __spirit__ is
fast. If you know the internals of __classic__ and perhaps
even wrote some parsers with it, this simple __spirit__ mechanism
is a joy to work with. There are no scanners and all that crap.
The /what/ function just tells us that it is an integer parser. Simple.
The /attribute/ metafunction returns the T template parameter. We associate the
`any_int_parser` to some placeholders for `short_`, `int_`, `long_` and
`long_long` types. But, first, we enable these placeholders in namespace
boost::spirit:
[primitive_parsers_enable_short]
[primitive_parsers_enable_int]
[primitive_parsers_enable_long]
[primitive_parsers_enable_long_long]
Notice that `any_int_parser` is placed in the namespace boost::spirit::qi
while these /enablers/ are in namespace boost::spirit. The reason is
that these placeholders are shared by other __spirit__ /domains/. __qi__,
the parser is one domain. __karma__, the generator is another domain.
Other parser technologies may be developed and placed in yet
another domain. Yet, all these can potentially share the same
placeholders for interoperability. The interpretation of these
placeholders is domain-specific.
Now that we enabled the placeholders, we have to write generators
for them. The make_xxx stuff (in boost::spirit::qi namespace):
[primitive_parsers_make_int]
This one above is our main generator. It's a simple function object
with 2 (unused) arguments. These arguments are
# The actual terminal value obtained by proto. In this case, either
a short_, int_, long_ or long_long. We don't care about this.
# Modifiers. We also don't care about this. This allows directives
such as `no_case[p]` to pass information to inner parser nodes.
We'll see how that works later.
Now:
[primitive_parsers_short_primitive]
[primitive_parsers_int_primitive]
[primitive_parsers_long_primitive]
[primitive_parsers_long_long_primitive]
These, specialize `qi:make_primitive` for specific tags. They all
inherit from `make_int` which does the actual work.
[heading Composite Parsers]
Let me present the kleene star (also in namespace spirit::qi):
[import ../../../../boost/spirit/home/qi/operator/kleene.hpp]
[composite_parsers_kleene]
Looks similar in form to its primitive cousin, the `int_parser`. And, again, it
has the same basic ingredients required by `Derived`.
* The nested attribute metafunction
* The parse member function
* The what member function
kleene is a composite parser. It is a parser that composes another
parser, its ["subject]. It is a `__unary_parser_concept__` and subclasses from it.
Like `__primitive_parser_concept__`, `__unary_parser_concept__<Derived>` derives
from `parser<Derived>`.
unary_parser<Derived>, has these expression requirements on Derived:
* p.subject -> subject parser ( ['p] is a __unary_parser_concept__ parser.)
* P::subject_type -> subject parser type ( ['P] is a __unary_parser_concept__ type.)
/parse/ is the main parser entry point. Since this is not a primitive
parser, we do not need to call `qi::skip(first, last, skipper)`. The
['subject], if it is a primitive, will do the pre-skip. If if it is
another composite parser, it will eventually call a primitive parser
somewhere down the line which will do the pre-skip. This makes it a
lot more efficient than __classic__. __classic__ puts the skipping business
into the so-called "scanner" which blindly attempts a pre-skip
every time we increment the iterator.
What is the /attribute/ of the kleene? In general, it is a `std::vector<T>`
where `T` is the attribute of the subject. There is a special case though.
If `T` is an `unused_type`, then the attribute of kleene is also `unused_type`.
`traits::build_std_vector` takes care of that minor detail.
So, let's parse. First, we need to provide a local attribute of for
the subject:
``
typename traits::attribute_of<Subject, Context>::type val;
``
`traits::attribute_of<Subject, Context>` simply calls the subject's
`struct attribute<Context>` nested metafunction.
/val/ starts out default initialized. This val is the one we'll
pass to the subject's parse function.
The kleene repeats indefinitely while the subject parser is
successful. On each successful parse, we `push_back` the parsed
attribute to the kleene's attribute, which is expected to be,
at the very least, compatible with a `std::vector`. In other words,
although we say that we want our attribute to be a `std::vector`,
we try to be more lenient than that. The caller of kleene's
parse may pass a different attribute type. For as long as it is
also a conforming STL container with `push_back`, we are ok. Here
is the kleene loop:
``
while (subject.parse(first, last, context, skipper, val))
{
// push the parsed value into our attribute
traits::push_back(attr, val);
traits::clear(val);
}
return true;
``
Take note that we didn't call attr.push_back(val). Instead, we
called a Spirit provided function:
``
traits::push_back(attr, val);
``
This is a recurring pattern. The reason why we do it this way is
because attr [*can] be `unused_type`. `traits::push_back` takes care
of that detail. The overload for unused_type is a no-op. Now, you
can imagine why __spirit__ is fast! The parsers are so simple and the
generated code is as efficient as a hand rolled loop. All these
parser compositions and recursive parse invocations are extensively
inlined by a modern C++ compiler. In the end, you get a tight loop
when you use the kleene. No more excess baggage. If the attribute
is unused, then there is no code generated for that. That's how
__spirit__ is designed.
The /what/ function simply wraps the output of the subject in a
"kleene[" ... "]".
Ok, now, like the `int_parser`, we have to hook our parser to the
_qi_ engine. Here's how we do it:
First, we enable the prefix star operator. In proto, it's called
the "dereference":
[composite_parsers_kleene_enable_]
This is done in namespace `boost::spirit` like its friend, the `use_terminal`
specialization for our `int_parser`. Obviously, we use /use_operator/ to
enable the dereference for the qi::domain.
Then, we need to write our generator (in namespace qi):
[composite_parsers_kleene_generator]
This essentially says; for all expressions of the form: `*p`, to build a kleene
parser. Elements is a __fusion__ sequence. For the kleene, which is a unary
operator, expect only one element in the sequence. That element is the subject
of the kleene.
We still don't care about the Modifiers. We'll see how the modifiers is
all about when we get to deep directives.
[endsect]
[endsect]

View File

@@ -0,0 +1,59 @@
[/==============================================================================
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2001-2011 Joel de Guzman
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[/------------------------------------------------------------------------------]
[section XXX]
[heading Description]
Description of XXX concept
[heading Refinement of]
[:Link to base concept]
[variablelist Notation
[[`xxx`] [An XXX]]
]
[heading Valid Expressions]
(For any XXX the following expressions must be valid:)
In addition to the requirements defined in _XXX-Basic_concept_, for any
XXX the following must be met:
[table
[[Expression] [Semantics] [Return type]]
[[`xxx`] [Semantics of `xxx`] [XXX]]
]
[heading Type Expressions]
[table
[[Expression] [Description]]
[[`XXX`] [Description of `XXX`]]
]
[heading Invariants]
For any XXX xxx the following invariants always hold:
[heading Precondition]
Prior to calling FOO the following preconditions should hold:
[heading Precondition]
Upon return from FOO the following postconditions should hold:
[heading Models]
Links to models of XXX concept
[endsect] [/ XXX Concept]

View File

@@ -0,0 +1,82 @@
[/==============================================================================
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2001-2011 Joel de Guzman
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[/////////////////////////////////////////////////////////////////////////////]
[section:Customization_Point XXX]
[heading Customization_Point]
Short description here...
[heading Module Headers]
#include <boost/spirit/xxx.hpp>
Also, see __include_structure__.
[note This header file does not need to be included directly by any user
program as it is normally included by other Spirit header files relying
on its content.]
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::xxx`]]
]
[heading Synopsis]
template <typename T>
struct XXX;
[heading Template parameters]
[table
[[Parameter] [Description] [Default]]
[[`T`] [What is T] []]
]
[variablelist Notation
[[`xxx`] [An XXX]]
]
[heading Expression Semantics]
[table
[[Expression] [Semantics]]
[[`xxx`] [Semantics of `xxx`]]
]
[heading Predefined Specializations]
[table
[[Type] [Semantics]]
[[`xxx`] [Summary of the specialization.]]
]
[heading When to implement]
Describe when this customization point needs to be implemented by the user.
[heading Related Attribute Customization Points]
If this customization point is implemented, the following other customization
points need to be implemented as well.
[table
[[Name] [When to implement]]
]
[heading Example]
Real example code. Use Quickbook import mechanism to link to actual
working code snippets here.
[endsect] [/ XXX]

View File

@@ -0,0 +1,162 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:faq Spirit FAQ]
[heading I'm getting multiple symbol definition errors while using Visual C++. Anything I could do about that?]
Do you see strange multiple symbol definition linker errors mentioning
`boost::mpl::failed` and `boost::spirit::qi::rule`? Then this FAQ entry might
be for you.
__mpl__ implements a macro `BOOST_MPL_ASSERT_MSG()` which essentially is a
more powerful version of static_assert. Unfortunately under certain
circumstances using this macro may lead to the aforementioned linker errors.
__spirit__ allows you to define a preprocessor constant disabling the usage
of `BOOST_MPL_ASSERT_MSG()`, while switching to `BOOST_STATIC_ASSERT()` instead.
For that you need define BOOST_SPIRIT_DONT_USE_MPL_ASSERT_MSG=1. Do this by
adding
-DBOOST_SPIRIT_DONT_USE_MPL_ASSERT_MSG=1
on the compiler command line or by inserting a
#define BOOST_SPIRIT_DONT_USE_MPL_ASSERT_MSG 1
into your code before any Spirit headers get included.
Using this trick has no adverse effects on any of the functionality of
__spirit__. The only change you might see while using this workaround is less
verbose error messages generated from static_assert.
[heading I'm very confused about the header hell in my boost/spirit directory. What's all this about?]
The boost/spirit directory currently holds two versions of the Spirit library:
__classic__ (former V1.8.x) and SpiritV2. Both are completely independent
and usually not used at the same time. Do not mix these two in the same grammar.
__classic__ evolved over years in a fairly complex directory structure:
boost/spirit/actor
boost/spirit/attribute
boost/spirit/core
boost/spirit/debug
boost/spirit/dynamic
boost/spirit/error_handling
boost/spirit/iterator
boost/spirit/meta
boost/spirit/symbols
boost/spirit/tree
boost/spirit/utility
While introducing Spirit V2 we restructured the directory structure in order to
accommodate two versions at the same time. All of __classic__ now lives in
the directory
boost/spirit/home/classic
where the directories above contain forwarding headers to the new location
allowing to maintain application compatibility. The forwarding headers issue a
warning (starting with Boost V1.38) telling the user to change their include
paths. Please expect the above directories/forwarding headers to go away soon.
This explains the need for the directory
boost/spirit/include
which contains forwarding headers as well. But this time the headers won't go
away. We encourage application writers to use only the includes contained in
this directory. This allows us to restructure the directories underneath if
needed without worrying application compatibility. Please use those files in
your application only. If it turns out that some forwarding file is missing,
please report this as a bug.
Spirit V2 is not about parsing only anymore (as __classic__). It now consists
out of 3 parts (sub-libraries): __qi__, __karma__, and __lex__. The header
files for those live in
boost/spirit/home/qi
boost/spirit/home/karma
boost/spirit/home/lex
and have forwarding headers in
boost/spirit/include
__qi__ is the direct successor to __classic__ as it implements a DSEL (domain
specific embedded language) allowing to write parsers using the syntax of C++
itself (parsers in the sense turning a sequence of bytes into an internal data
structure). It is not compatible with __classic__, the main concepts are
similar, though.
__karma__ is the counterpart to __qi__. It implements a similar DSEL but for
writing generators (i.e. the things turning internal data structures into a
sequence of bytes, most of the time - strings). __karma__ is the Yang to
__qi__'s Yin, it's almost like a mirrored picture.
__lex__ is (as the name implies) a library allowing to write lexical analyzers.
These are either usable stand alone or can be used as a front end for __qi__
parsers. If you know flex you shouldn't have problems understanding __lex__.
This library actually doesn't implement the lexer itself. All it does is to
provide an interface to pre-existing lexical analyzers. Currently it's using
Ben Hansons excellent __lexertl__ library (proposed for a Boost review, BTW) as
its underlying workhorse.
Again, don't use any of the header files underneath the boost/spirit/home
directory directly, always include files from the boost/spirit/include
directory.
[heading Why doesn't my symbol table work in a `no_case` directive?]
In order to perform case-insensitive parsing (using __qi_no_case__) with a
symbol table (i.e. use a __qi_symbols__
parser in a `no_case` directive), that symbol table needs to be filled with
all-lowercase contents. Entries containing one or more uppercase characters
will not match any input.
[heading I'm getting a compilation error mentioning `boost::function` and/or
`boost::function4`. What does this mean?]
If you are using Visual C++ and have an error like:
[pre
error C2664: \'bool boost::function4<R,T0,T1,T2,T3>::operator ()(T0,T1,T2,T3) const\' :
cannot convert parameter 4 from '...' to '...'
]
or you are using GCC and have an error like:
[pre
error: no match for call to '(const boost::function<bool ()(...)>) (...)'
note: candidates are: ... boost::function4<R,T1,T2,T3,T4>::operator()(T0,T1,T2,T3) const [with ...\]
]
then this FAQ entry may help you.
The definition of a __rule__ or __grammar__ may contain a skip parser type. If
it does, it means that non-terminal can only be used with a skip parser of a
compatible type. The error above arises when this is not the case, i.e.:
* a non-terminal defined with a skip parser type is used without a skip parser;
for example, a rule with a skip parser type is used inside a `lexeme`
directive, or a grammar with a skip parser type is used in `parse` instead of
`phrase_parse`,
* or a non-terminal is used with a skip parser of an incompatible type;
for example, a rule defined with one skip parser type calls a second rule
defined with another, incompatible skip parser type.
[note The same applies to __karma__, replacing 'skip parser' and `lexeme`
by 'delimit generator' and `verbatim`. Similarly, corresponding error
messages in __karma__ reference `boost::function3` and the 3rd
parameter (instead of the 4th).]
[endsect]

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.5 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 391 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 485 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 410 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 488 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 509 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 499 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 507 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 446 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 431 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 441 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 423 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 431 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 397 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 434 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 420 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 61 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 78 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 59 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 83 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.6 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.8 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 105 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 107 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.6 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.5 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

View File

@@ -0,0 +1,16 @@
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<!-- Copyright (C) 2002 Douglas Gregor <doug.gregor -at- gmail.com>
Distributed under the Boost Software License, Version 1.0.
(See accompanying file LICENSE_1_0.txt or copy at
http://www.boost.org/LICENSE_1_0.txt) -->
<title>Redirect to generated documentation</title>
<meta http-equiv="refresh" content="0; URL=http://boost-sandbox.sourceforge.net/libs/spirit/doc/html/">
</head>
<body>
Automatic redirection failed, please go to
<a href="http://boost-sandbox.sourceforge.net/libs/spirit/doc/html/">http://boost-sandbox.sourceforge.net/libs/spirit/doc/html/</a>
</body>
</html>

View File

@@ -0,0 +1,288 @@
#==============================================================================
# Copyright (c) 2001-2011 Hartmut Kaiser
#
# Use, modification and distribution is subject to the Boost Software
# License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at
# http://www.boost.org/LICENSE_1_0.txt)
#==============================================================================
#!scan-path "../../../boost/spirit/home/qi" ".*hpp" true
#!scan-path "../../../boost/spirit/home/karma" ".*hpp" true
###############################################################################
# Qi API
parse "" ".*qi.reference.parse_api.iterator_api.*" qi_index
phrase_parse "" ".*qi.reference.parse_api.iterator_api.*" qi_index
match "" ".*qi.reference.parse_api.stream_api.*" qi_index
phrase_match "" ".*qi.reference.parse_api.stream_api.*" qi_index
skip_flag::postskip "postskip" ".*qi.reference.parse_api.*_api.*" qi_index
skip_flag::dont_postskip "dont_postskip" ".*qi.reference.parse_api.*_api.*" qi_index
create_parser "" ".*qi.reference.parse_api.*" qi_index
create_parser_exists "" ".*qi.reference.parse_api.*" qi_index
parser "[P|p]arser" ".*qi.reference.parser_concepts.*" qi_index
###############################################################################
# Karma API
generate "" ".*karma.reference.generate_api.iterator_api.*" karma_index
generate_delimited "" ".*karma.reference.generate_api.iterator_api.*" karma_index
format "" ".*karma.reference.generate_api.stream_api.*" karma_index
format_delimited "" ".*karma.reference.generate_api.stream_api.*" karma_index
delimit_flag::pre_delimit "pre_delimit" ".*karma.reference.generate_api.*_api.*" karma_index
delimit_flag::dont_predelimit "dont_predelimit" ".*karma.reference.generate_api.*_api.*" karma_index
create_generator "" ".*karma.reference.generate_api.*" karma_index
create_generator_exists "" ".*karma.reference.generate_api.*" karma_index
generator "[G|g]enerator" ".*karma.reference.generator_concepts.*" karma_index
###############################################################################
# auto parser/generator
auto_ "" ".*qi.reference.auto.*" qi_index
auto_ "" ".*karma.reference.auto.*" karma_index
# auxiliary parsers/generators
attr "" ".*qi.reference.auxiliary.attr" qi_index
attr_cast "" ".*qi.reference.auxiliary.attr_cast.*" qi_index
eol "" ".*qi.reference.auxiliary.eol.*" qi_index
eoi "" ".*qi.reference.auxiliary.eoi.*" qi_index
eps "" ".*qi.reference.auxiliary.eps.*" qi_index
lazy "" ".*qi.reference.auxiliary.lazy.*" qi_index
attr_cast "" ".*karma.reference.auxiliary.attr_cast.*" karma_index
eol "" ".*karma.reference.auxiliary.eol.*" karma_index
eoi "" ".*karma.reference.auxiliary.eoi.*" karma_index
eps "" ".*karma.reference.auxiliary.eps.*" karma_index
lazy "" ".*karma.reference.auxiliary.lazy.*" karma_index
# binary parsers/generators
byte_ "" ".*qi.reference.binary.binary_native.*" qi_index
word "" ".*qi.reference.binary.binary_native.*" qi_index
dword "" ".*qi.reference.binary.binary_native.*" qi_index
qword "" ".*qi.reference.binary.binary_native.*" qi_index
bin_float "" ".*qi.reference.binary.binary_native.*" qi_index
bin_double "" ".*qi.reference.binary.binary_native.*" qi_index
little_word "" ".*qi.reference.binary.binary_little.*" qi_index
little_dword "" ".*qi.reference.binary.binary_little.*" qi_index
little_qword "" ".*qi.reference.binary.binary_little.*" qi_index
little_bin_float "" ".*qi.reference.binary.binary_little.*" qi_index
little_bin_double "" ".*qi.reference.binary.binary_little.*" qi_index
big_word "" ".*qi.reference.binary.binary_big.*" qi_index
big_dword "" ".*qi.reference.binary.binary_big.*" qi_index
big_qword "" ".*qi.reference.binary.binary_big.*" qi_index
big_bin_float "" ".*qi.reference.binary.binary_big.*" qi_index
big_bin_double "" ".*qi.reference.binary.binary_big.*" qi_index
byte_ "" ".*karma.reference.binary.binary_native.*" karma_index
word "" ".*karma.reference.binary.binary_native.*" karma_index
dword "" ".*karma.reference.binary.binary_native.*" karma_index
qword "" ".*karma.reference.binary.binary_native.*" karma_index
bin_float "" ".*karma.reference.binary.binary_native.*" karma_index
bin_double "" ".*karma.reference.binary.binary_native.*" karma_index
little_word "" ".*karma.reference.binary.binary_little.*" karma_index
little_dword "" ".*karma.reference.binary.binary_little.*" karma_index
little_qword "" ".*karma.reference.binary.binary_little.*" karma_index
little_bin_float "" ".*karma.reference.binary.binary_little.*" karma_index
little_bin_double "" ".*karma.reference.binary.binary_little.*" karma_index
big_word "" ".*karma.reference.binary.binary_big.*" karma_index
big_dword "" ".*karma.reference.binary.binary_big.*" karma_index
big_qword "" ".*karma.reference.binary.binary_big.*" karma_index
big_bin_float "" ".*karma.reference.binary.binary_big.*" karma_index
big_bin_double "" ".*karma.reference.binary.binary_big.*" karma_index
# char parsers/generators
char_ "" ".*qi.reference.char\..*" qi_index
lit "" "((.*qi.reference.(char|string))|(.*qi.reference.numeric)).*" qi_index
alnum "" ".*qi.reference.char.char_.*" qi_index
alpha "" ".*qi.reference.char.char_.*" qi_index
blank "" ".*qi.reference.char.char_.*" qi_index
cntrl "" ".*qi.reference.char.char_.*" qi_index
digit "" ".*qi.reference.char.char_.*" qi_index
graph "" ".*qi.reference.char.char_.*" qi_index
print "" ".*qi.reference.char.char_.*" qi_index
punct "" ".*qi.reference.char.char_class.*" qi_index
space "" ".*qi.reference.char.char_class.*" qi_index
xdigit "" ".*qi.reference.char.char_.*" qi_index
lower "" ".*qi.reference.char.char_class.*" qi_index
upper "" ".*qi.reference.char.char_class.*" qi_index
char_ "" ".*karma.reference.char\..*" karma_index
lit "" "((.*karma.reference.(char|string))|(.*karma.reference.numeric)).*" karma_index
alnum "" ".*karma.reference.char.char_.*" karma_index
alpha "" ".*karma.reference.char.char_.*" karma_index
blank "" ".*karma.reference.char.char_.*" karma_index
cntrl "" ".*karma.reference.char.char_.*" karma_index
digit "" ".*karma.reference.char.char_.*" karma_index
graph "" ".*karma.reference.char.char_.*" karma_index
print "" ".*karma.reference.char.char_.*" karma_index
punct "" ".*karma.reference.char.char_class.*" karma_index
space "" ".*karma.reference.char.char_class.*" karma_index
xdigit "" ".*karma.reference.char.char_.*" karma_index
lower "" ".*karma.reference.((char.char_class.*)|(directive\..*))" karma_index
upper "" ".*karma.reference.((char.char_class.*)|(directive\..*))" karma_index
# directives
lexeme "" ".*qi.reference.directive\..*" qi_index
no_skip "" ".*qi.reference.directive\..*" qi_index
no_case "" ".*qi.reference.directive\..*" qi_index
raw "" ".*qi.reference.directive\..*" qi_index
matches "" ".*qi.reference.directive\..*" qi_index
skip "" ".*qi.reference.directive\..*" qi_index
repeat "" ".*qi.reference.directive\..*" qi_index
inf "" ".*qi.reference.directive\..*" qi_index
omit "" ".*qi.reference.directive\..*" qi_index
"as<T>" "as" ".*qi.reference.directive.as" qi_index
as_string "" ".*qi.reference.directive.as" qi_index
as_wstring "" ".*qi.reference.directive.as" qi_index
hold "" ".*qi.reference.directive.hold" qi_index
expect "" ".*qi.reference.directive\..*" qi_index
left_align "" ".*karma.reference.directive.alignment.*" karma_index
center "" ".*karma.reference.directive.alignment.*" karma_index
right_align "" ".*karma.reference.directive.alignment.*" karma_index
verbatim "" ".*karma.reference.directive\..*" karma_index
delimit "" ".*karma.reference.directive\..*" karma_index
no_delimit "" ".*karma.reference.directive\..*" karma_index
maxwidth "" ".*karma.reference.directive\..*" karma_index
buffer "" ".*karma.reference.directive\..*" karma_index
duplicate "" ".*karma.reference.directive\..*" karma_index
columns "" ".*karma.reference.directive\..*" karma_index
repeat "" ".*karma.reference.directive\..*" karma_index
inf "" ".*karma.reference.directive\..*" karma_index
omit "" ".*karma.reference.directive\..*" karma_index
"as<T>" "as" ".*karma.reference.directive.as" karma_index
as_string "" ".*karma.reference.directive.as" karma_index
as_wstring "" ".*karma.reference.directive.as" karma_index
# nonterminal parsers/generators
rule "" ".*qi.reference.nonterminal.rule.*" qi_index
grammar "" ".*qi.reference.nonterminal.grammar.*" qi_index
rule "" ".*karma.reference.nonterminal.rule.*" karma_index
grammar "" ".*karma.reference.nonterminal.grammar.*" karma_index
# numeric parsers/generators
int_ "" ".*qi.reference.numeric\..*" qi_index
bin "" ".*qi.reference.numeric\..*" qi_index
oct "" ".*qi.reference.numeric\..*" qi_index
hex "" ".*qi.reference.numeric\..*" qi_index
ushort_ "" ".*qi.reference.numeric\..*" qi_index
uint_ "" ".*qi.reference.numeric\..*" qi_index
ulong_ "" ".*qi.reference.numeric\..*" qi_index
ulong_long "" ".*qi.reference.numeric\..*" qi_index
short_ "" ".*qi.reference.numeric\..*" qi_index
int_ "" ".*qi.reference.numeric\..*" qi_index
long_ "" ".*qi.reference.numeric\..*" qi_index
long_long "" ".*qi.reference.numeric\..*" qi_index
float_ "" ".*qi.reference.numeric\..*" qi_index
double_ "" ".*qi.reference.numeric\..*" qi_index
long_double "" ".*qi.reference.numeric\..*" qi_index
bool_ "" ".*qi.reference.numeric\..*" qi_index
true_ "" ".*qi.reference.numeric\..*" qi_index
false_ "" ".*qi.reference.numeric\..*" qi_index
int_ "" ".*karma.reference.numeric\..*" karma_index
bin "" ".*karma.reference.numeric\..*" karma_index
oct "" ".*karma.reference.numeric\..*" karma_index
hex "" ".*karma.reference.numeric\..*" karma_index
ushort_ "" ".*karma.reference.numeric\..*" karma_index
uint_ "" ".*karma.reference.numeric\..*" karma_index
ulong_ "" ".*karma.reference.numeric\..*" karma_index
ulong_long "" ".*karma.reference.numeric\..*" karma_index
short_ "" ".*karma.reference.numeric\..*" karma_index
int_ "" ".*karma.reference.numeric\..*" karma_index
long_ "" ".*karma.reference.numeric\..*" karma_index
long_long "" ".*karma.reference.numeric\..*" karma_index
float_ "" ".*karma.reference.numeric\..*" karma_index
double_ "" ".*karma.reference.numeric\..*" karma_index
long_double "" ".*karma.reference.numeric\..*" karma_index
bool_ "" ".*karma.reference.numeric\..*" karma_index
true_ "" ".*karma.reference.numeric\..*" karma_index
false_ "" ".*karma.reference.numeric\..*" karma_index
bool_policies "" ".*qi.reference.numeric\..*" qi_index
real_policies "" ".*qi.reference.numeric\..*" qi_index
ureal_policies "" ".*qi.reference.numeric\..*" qi_index
bool_policies "" ".*karma.reference.numeric\..*" karma_index
real_policies "" ".*karma.reference.numeric\..*" karma_index
ureal_policies "" ".*karma.reference.numeric\..*" karma_index
bool_parser "" ".*qi.reference.numeric\..*" qi_index
int_parser "" ".*qi.reference.numeric\..*" qi_index
uint_parser "" ".*qi.reference.numeric\..*" qi_index
real_parser "" ".*qi.reference.numeric\..*" qi_index
strict_real_parser "" ".*qi.reference.numeric\..*" qi_index
bool_generator "" ".*karma.reference.numeric\..*" karma_index
int_generator "" ".*karma.reference.numeric\..*" karma_index
uint_generator "" ".*karma.reference.numeric\..*" karma_index
real_generator "" ".*karma.reference.numeric\..*" karma_index
# parser/generator operators
"sequence (a >> b)" "sequence" ".*qi.reference.operator.sequence.*" qi_index
"alternative (a | b)" "alternative" ".*qi.reference.operator.alternative.*" qi_index
"kleene (*a)" "[K|k]leene" ".*qi.reference.operator.kleene.*" qi_index
"plus (+a)" "[P|p]lus" ".*qi.reference.operator.plus.*" qi_index
"list (a % b)" "list" ".*qi.reference.operator.list.*" qi_index
"optional (-a)" "optional" ".*qi.reference.operator.optional.*" qi_index
"and-predicate (&a)" "and-predicate" ".*qi.reference.operator.and_predicate.*" qi_index
"not-predicate (!a)" "not-predicate" ".*qi.reference.operator.not_predicate.*" qi_index
"expectation (a > b)" "expectation" ".*qi.reference.operator.expect.*" qi_index
"difference (a - b)" "difference" ".*qi.reference.operator.difference.*" qi_index
"permutation (a ^ b)" "permutation" ".*qi.reference.operator.permutation.*" qi_index
"sequential-or (a || b)" "sequential-or" ".*qi.reference.operator.sequential_or.*" qi_index
"sequence (a << b)" "sequence" ".*karma.reference.operator.sequence.*" karma_index
"alternative (a | b)" "alternative" ".*karma.reference.operator.alternative.*" karma_index
"kleene (*a)" "[K|k]leene" ".*karma.reference.operator.kleene.*" karma_index
"plus (+a)" "[P|p]lus" ".*karma.reference.operator.plus.*" karma_index
"list (a % b)" "list" ".*karma.reference.operator.list.*" karma_index
"optional (-a)" "optional" ".*karma.reference.operator.optional.*" karma_index
"and-predicate (&a)" "and-predicate" ".*karma.reference.operator.and_predicate.*" karma_index
"not-predicate (!a)" "not-predicate" ".*karma.reference.operator.not_predicate.*" karma_index
# stream parsers/generators
stream "" ".*qi.reference.operator.stream\..*" qi_index
wstream "" ".*qi.reference.operator.stream\..*" qi_index
stream_parser "" ".*qi.reference.operator.stream\..*" qi_index
stream "" ".*karma.reference.operator.stream\..*" karma_index
wstream "" ".*karma.reference.operator.stream\..*" karma_index
stream_generator "" ".*karma.reference.operator.stream\..*" karma_index
# string parsers/generators
string "" ".*qi.reference.string\..*" qi_index
symbols "" ".*qi.reference.string\..*" qi_index
string "" ".*karma.reference.string\..*" karma_index
symbols "" ".*karma.reference.string\..*" karma_index
# semantic actions
"Semantic Action" "[A|a]ctions+" ".*(qi.reference|turorials).(action|semantic_actions).*" qi_index
"Semantic Action" "[A|a]ctions+" ".*(karma.reference|turorials).(action|semantic_actions).*" karma_index
# placeholders
placeholders "_\d|_r\d|_val|_[a-j]|_pass" ".*qi.quick_reference.phoenix.*" qi_index
"_1 ... _N" "_\d" ".*qi.quick_reference.phoenix.*" qi_index
"_r1 ... _rN" "_r\d" ".*qi.quick_reference.phoenix.*" qi_index
_val "" ".*qi.quick_reference.phoenix.*" qi_index
"_a ... _j" "_[a-j]" ".*qi.quick_reference.phoenix.*" qi_index
_pass "" ".*qi.quick_reference.phoenix.*" qi_index
placeholders "_\d|_r\d|_val|_[a-j]|_pass" ".*karma.quick_reference.phoenix.*" karma_index
"_1 ... _N" "_\d" ".*karma.quick_reference.phoenix.*" karma_index
"_r1 ... _rN" "_r\d" ".*karma.quick_reference.phoenix.*" karma_index
_val "" ".*karma.quick_reference.phoenix.*" karma_index
"_a ... _j" "_[a-j]" ".*karma.quick_reference.phoenix.*" karma_index
_pass "" ".*karma.quick_reference.phoenix.*" karma_index
###############################################################################
#!exclude N
#!exclude type iterator Auto call where: f info derived_type subject_type
#!exclude if floatfield precision trailing_zeros force_sign in pointer
#!exclude result_type value_type difference_type assign clear

View File

@@ -0,0 +1,205 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section Introduction]
Boost Spirit is an object-oriented, recursive-descent parser and
output generation library for C++. It allows you to write grammars and
format descriptions using a format similar to Extended Backus Naur
Form (EBNF)[footnote [@http://www.cl.cam.ac.uk/%7Emgk25/iso-14977.pdf
ISO-EBNF]] directly in C++. These inline grammar
specifications can mix freely with other C++ code and, thanks to the
generative power of C++ templates, are immediately executable. In
retrospect, conventional compiler-compilers or parser-generators have
to perform an additional translation step from the source EBNF code to
C or C++ code.
The syntax and semantics of the libraries' API directly form domain-specific
embedded languages (DSEL). In fact, Spirit exposes 3 different DSELs to the
user:
* one for creating parser grammars,
* one for the specification of the required tokens to be used for parsing,
* and one for the description of the required output formats.
Since the target input grammars and output formats are written entirely in C++
we do not need any separate tools to compile, preprocess or integrate those
into the build process. __spirit__ allows seamless integration of the parsing
and output generation process with other C++ code. This often allows for
simpler and more efficient code.
Both the created parsers and generators are fully attributed, which allows you
to easily build and handle hierarchical data structures in memory. These data
structures resemble the structure of the input data and can directly be used
to generate arbitrarily-formatted output.
The [link spirit.spiritstructure figure] below depicts the overall structure
of the Boost Spirit library. The library consists of 4 major parts:
* __classic__: This is the almost-unchanged code base taken from the
former Boost Spirit V1.8 distribution. It has been moved into the namespace
boost::spirit::classic. A special compatibility layer has been added to
ensure complete compatibility with existing code using Spirit V1.8.
* __qi__: This is the parser library allowing you to build recursive
descent parsers. The exposed domain-specific language can be used to describe
the grammars to implement, and the rules for storing the parsed information.
* __lex__: This is the library usable to create tokenizers (lexers). The
domain-specific language exposed by __lex__ allows you to define regular
expressions used to match tokens (create token definitions), associate these
regular expressions with code to be executed whenever they are matched, and
to add the token definitions to the lexical analyzer.
* __karma__: This is the generator library allowing you to create code for
recursive descent, data type-driven output formatting. The exposed
domain-specific language is almost equivalent to the parser description language
used in __qi__, except that it is used to describe the required output
format to generate from a given data structure.
[fig spiritstructure.png..The overall structure of the Boost Spirit library..spirit.spiritstructure]
The three components, __qi__, __karma__ and __lex__, are designed to be used
either stand alone, or together. The general methodology is to use the token
sequence generated by __lex__ as the input for a parser generated by __qi__.
On the opposite side of the equation, the hierarchical data structures generated
by __qi__ are used for the output generators created using __karma__.
However, there is nothing to stop you from using any of these components all
by themselves.
The [link spirit.spiritkarmaflow figure] below shows the typical data flow of
some input being converted to some internal representation. After some
(optional) transformation these data are converted back into some different,
external representation. The picture highlights Spirit's place in this data
transformation flow.
[fig spiritkarmaflow.png..The place of __qi__ and __karma__ in a data transformation flow of a typical application..spirit.spiritkarmaflow]
[heading A Quick Overview of Parsing with __qi__]
__qi__ is Spirit's sublibrary dealing with generating parsers based on a given
target grammar (essentially a format description of the input data to read).
A simple EBNF grammar snippet:
group ::= '(' expression ')'
factor ::= integer | group
term ::= factor (('*' factor) | ('/' factor))*
expression ::= term (('+' term) | ('-' term))*
is approximated using facilities of Spirit's /Qi/ sublibrary as seen in this
code snippet:
group = '(' >> expression >> ')';
factor = integer | group;
term = factor >> *(('*' >> factor) | ('/' >> factor));
expression = term >> *(('+' >> term) | ('-' >> term));
Through the magic of expression templates, this is perfectly valid and
executable C++ code. The production rule `expression` is, in fact, an object that
has a member function `parse` that does the work given a source code written in
the grammar that we have just declared. Yes, it's a calculator. We shall
simplify for now by skipping the type declarations and the definition of the
rule `integer` invoked by `factor`. Now, the production rule `expression` in our
grammar specification, traditionally called the `start` symbol, can recognize
inputs such as:
12345
-12345
+12345
1 + 2
1 * 2
1/2 + 3/4
1 + 2 + 3 + 4
1 * 2 * 3 * 4
(1 + 2) * (3 + 4)
(-1 + 2) * (3 + -4)
1 + ((6 * 200) - 20) / 6
(1 + (2 + (3 + (4 + 5))))
Certainly we have modified the original EBNF syntax. This is done to
conform to C++ syntax rules. Most notably we see the abundance of
shift >> operators. Since there are no 'empty' operators in C++, it is
simply not possible to write something like:
a b
as seen in math syntax, for example, to mean multiplication or, in our case,
as seen in EBNF syntax to mean sequencing (b should follow a). __qi__
uses the shift `>>` operator instead for this purpose. We take the `>>` operator,
with arrows pointing to the right, to mean "is followed by". Thus we write:
a >> b
The alternative operator `|` and the parentheses `()` remain as is. The
assignment operator `=` is used in place of EBNF's `::=`. Last but not least,
the Kleene star `*`, which in this case is a postfix operator in EBNF becomes a
prefix. Instead of:
a* //... in EBNF syntax,
we write:
*a //... in Spirit.
since there are no postfix stars, `*`, in C/C++. Finally, we terminate each
rule with the ubiquitous semi-colon, `;`.
[heading A Quick Overview of Output Generation with __karma__]
Spirit not only allows you to describe the structure of the input, it also enables
the specification of the output format for your data in a similar way, and based
on a single syntax and compatible semantics.
Let's assume we need to generate a textual representation from a simple data
structure such as a `std::vector<int>`. Conventional code probably would look like:
std::vector<int> v (initialize_and_fill());
std::vector<int>::iterator end = v.end();
for (std::vector<int>::iterator it = v.begin(); it != end; ++it)
std::cout << *it << std::endl;
which is not very flexible and quite difficult to maintain when it comes to
changing the required output format. Spirit's sublibrary /Karma/ allows you to
specify output formats for arbitrary data structures in a very flexible way.
The following snippet is the /Karma/ format description used to create the
same output as the traditional code above:
*(int_ << eol)
Here are some more examples of format descriptions for different output
representations of the same `std::vector<int>`:
[table Different output formats for `std::vector<int>`
[ [Format] [Example] [Description] ]
[ [`'[' << *(int_ << ',') << ']'`] [`[1,8,10,]`] [Comma separated list of integers] ]
[ [`*('(' << int_ << ')' << ',')`] [`(1),(8),(10),`] [Comma separated list of integers in parenthesis] ]
[ [`*hex`] [`18a`] [A list of hexadecimal numbers] ]
[ [`*(double_ << ',')`] [`1.0,8.0,10.0,`] [A list of floating point numbers] ]
]
We will see later in this documentation how it is possible to avoid printing
the trailing `','`.
Overall, the syntax is similar to __qi__ with the exception that we use the `<<`
operator for output concatenation. This should be easy to understand as it
follows the conventions used in the Standard's I/O streams.
Another important feature of __karma__ allows you to fully decouple the data
type from the output format. You can use the same output format with different
data types as long as these conform conceptually. The next table gives some
related examples.
[table Different data types usable with the output format `*(int_ << eol)`
[ [Data type] [Description] ]
[ [`int i[4]`] [C style arrays] ]
[ [`std::vector<int>`] [Standard vector] ]
[ [`std::list<int>`] [Standard list] ]
[ [`boost::array<long, 20>`] [Boost array] ]
]
[endsect]

View File

@@ -0,0 +1,51 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:karma Karma - Writing Generators]
[section Tutorials]
[include karma/tutorial_intro.qbk]
[include karma/warming_up.qbk]
[include karma/actions.qbk]
[include karma/complex.qbk]
[include karma/num_list.qbk]
[endsect]
[section Quick Reference]
[include karma/quick_reference.qbk]
[endsect]
[section Reference]
[import ../example/karma/reference.cpp]
[include karma/concepts.qbk]
[include karma/basics.qbk]
[include karma/generate_api.qbk]
[include karma/action.qbk]
[include karma/auto.qbk]
[include karma/auxiliary.qbk]
[include karma/binary.qbk]
[include karma/char.qbk]
[include karma/directive.qbk]
[include karma/nonterminal.qbk]
[include karma/numeric.qbk]
[include karma/operator.qbk]
[include karma/stream.qbk]
[include karma/string.qbk]
[/ include karma/debug.qbk]
[endsect]
[section Performance Measurements]
[include karma/numeric_performance.qbk]
[endsect]
[? __use_auto_index__
[index Karma Index..karma_index]
]
[endsect]

View File

@@ -0,0 +1,119 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:action Semantic Actions with Generators]
[heading Description]
Semantic actions may be attached to any point in the grammar specification.
They allow to call a function or function object in order to provide the value
output by the generator attached to the semantic action. Semantic
actions are associated with a generator using the syntax `g[]`, where `g` is an
arbitrary generator expression.
[heading Header]
// forwards to <boost/spirit/home/karma/action.hpp>
#include <boost/spirit/include/karma_action.hpp>
Also, see __include_structure__.
[heading Model of]
[:__unary_generator_concept__]
[variablelist Notation
[[`a`, `g`][Instances of a generator, `G`]]
[[`A`] [Attribute type exposed by a generator, `a`]]
[[`fa`] [A (semantic action) function with signature `void(Attrib&, Context&, bool&)`.
The third parameter is a boolean flag that can be set to false to
force the generator to fail. Both `Context` and the boolean flag are
optional. For more information see below.]]
[[`Attrib`][The attribute to be used to generate output from.]]
[[`Context`] [The type of the generator execution context. For more
information see below.]]
]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is not
defined in __unary_generator_concept__.
[table
[[Expression] [Semantics]]
[[`g[fa]`] [Call semantic action, `fa` /before/ invoking `g`. The function
or function object `fa` is expected to provide the value
to generate output from to the generator `g`.]]
]
The possible signatures for functions to be used as semantic actions are:
template <typename Attrib>
void fa(Attrib& attr);
template <typename Attrib, typename Context>
void fa(Attrib& attr, Context& context);
template <typename Attrib, typename Context>
void fa(Attrib& attr, Context& context, bool& pass);
The function or function object is expected to return the value to generate
output from by assigning it to the first parameter, `attr`. Here `Attrib` is
the attribute type of the generator attached to the semantic action.
The type `Context` is the type of the generator execution context. This type is
unspecified and depends on the context the generator is invoked in. The value
`context` is used by semantic actions written using __phoenix__ to access various
context dependent attributes and values. For more information about __phoenix__
placeholder expressions usable in semantic actions see __karma_nonterminal_concept__.
The third parameter, `pass`, can be used by the semantic action to force the
associated generator to fail. If pass is set to `false` the action generator
will immediately return `false` as well, while not invoking `g` and not
generating any output.
[heading Attributes]
[table
[[Expression] [Attribute]]
[[`a[fa]`] [`a: A --> a[fa]: A`]]
]
[heading Complexity]
The complexity of the action generator is defined by the complexity of the
generator the semantic action is attached to and the complexity of the function
or function object used as the semantic action.
[important Please note that the use of semantic actions in __karma__ generally
forces the library to create a /copy/ of the attribute, which might
be a costly operation. Always consider using other means of
associating a value with a particular generator first.]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_action]
Some examples:
[reference_karma_action]
More examples for semantic actions can be found here:
[link spirit.karma.tutorials.semantic_actions.examples_of_semantic_actions Examples of Semantic Actions].
[endsect] [/ Action]

View File

@@ -0,0 +1,155 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:semantic_actions Generator Semantic Actions]
In the previous section we mentioned a very important difference between parsers
and generators. While parsers may be used without 'producing' any data,
generators always need data to generate the output from. We mentioned one way
of passing data to the generator by supplying it as a parameter to one of the
main API functions (for instance `generate()` or `generate_delimited()`).
But sometimes this is not possible or not desirable.
Very much like for __qi__ we have semantic actions in __karma__ as well.
Semantic actions may be attached to any point in the grammar specification.
These actions are C++ functions or function objects that are called whenever a
part of the generator is about to be invoked. Say you have a generator `G`,
and a C++ function `F`, you can make the generator call `F` just before it gets
invoked by attaching `F`:
G[F]
The expression above links `F` to the generator, `G`.
Semantic actions in __qi__ are invoked after a parser successfully
matches its input and the matched value is passed into the
semantic action. In __karma__ the opposite happens. Semantic actions are called
before its associated generator is invoked. They may provide the data required
by the generator.
The function/function object signature depends on the type of the generator to
which it is attached. The generator `double_` expects the number to generate.
Thus, if we were to attach a function `F` to `double_`, we need `F` to be
declared as:
void F(double& n);
where the function is expected to initialize the parameter `n` with the value
to generate.
[important Generally, and more formally, the semantic action `F` attached to
a generator `G` needs to take a reference to the generator's
attribute type as its first parameter. For more information about
generator attributes please see the section __karma_attribute__.
In the example above the function F takes a `double&` as its first
parameter as the attribute of the `double_` generator happens to be
a `double`.
]
There are actually 2 more arguments being passed (the generator context and a
reference to a boolean 'pass' parameter). We don't need these, for now, but
we'll see more on these other arguments later. __karma__ allows us to bind a
single argument function, like above. The other arguments are simply ignored.
To sum up, the possible signatures for semantic actions are:
void f(Attrib&);
void f(Attrib&, Context&);
void f(Attrib&, Context&, bool&);
[heading Examples of Semantic Actions]
In the following example we present various ways to attach semantic actions:
* Using a plain function pointer
* Using a simple function object
* Using __boost_bind__ with a plain function
* Using __boost_bind__ with a member function
* Using __boost_lambda__
[import ../../example/karma/actions.cpp]
Let's assume we have:
[karma_tutorial_semantic_action_functions]
Take note that with function objects, we need to have an `operator()` with 3
arguments. Since we don't care about the other two, we can use `unused_type` for
these. We'll see more of `unused_type` elsewhere. Get used to it. `unused_type`
is a Spirit supplied support class. Most of the time it stands for 'I don't
care, just use the appropriate default'.
All following examples generate outputs of the form:
"{integer}"
An integer inside the curly braces.
The first example shows how to attach a plain function:
[karma_tutorial_attach_actions1]
What's new? Well `int_` is the sibling of `double_`. I'm sure you can guess
what this generator does and what type of attribute it expects.
The next example shows how to attach a simple function object:
[karma_tutorial_attach_actions2]
We can use __boost_bind__ to 'bind' member functions:
[karma_tutorial_attach_actions4]
Likewise, we can also use __boost_bind__ to 'bind' plain functions:
[karma_tutorial_attach_actions3]
And last but not least, we can also use __boost_lambda__:
[karma_tutorial_attach_actions5]
There are more ways to bind semantic action functions, but the examples above
are the most common. Attaching semantic actions is the first hurdle one has
to tackle when getting started with generating with Spirit. If you didn't do so
yet, it is probably a good idea to familiarize yourself with the tools behind
it such as __boost_bind__ and __boost_lambda__.
The examples above can be found here: [@../../example/karma/actions.cpp actions.cpp]
[heading Phoenix]
__phoenix__, a companion library bundled with Spirit, is specifically suited
for binding semantic actions. It is like __boost_lambda__ on steroids, with
special custom features that make it easy to integrate semantic actions with
Spirit. If your requirements go beyond simple to moderate generation, I suggest
you use this library. Examples presented henceforth shall be using the Phoenix
library exclusively.
[important There are different ways to write semantic actions for __karma__:
using plain functions, __boost_bind__, __boost_lambda__, or
__phoenix__. The latter three allow you to use special placeholders
to control parameter placement (`_1`, `_2`, etc.). Each of those
libraries has it's own implementation of the placeholders, all
in different namespaces. You have to make sure not to mix
placeholders with a library they don't belong to and not to
use different libraries while writing a semantic action.
Generally, for __boost_bind__, use `::_1`, `::_2`, etc. (yes, these
placeholders are defined in the global namespace).
For __boost_lambda__ use the placeholders defined in the namespace
`boost::lambda`.
For semantic actions written using __phoenix__ use the placeholders
defined in the namespace `boost::spirit`. Please note that all
existing placeholders for your convenience are also available from
the namespace `boost::spirit::karma`.]
[endsect] [/ Semantic Actions]

View File

@@ -0,0 +1,145 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:auto Auto Generator]
[heading Description]
This module includes the description of the `auto_` generator. This generator
can be used to automatically create a generator based on the supplied attribute
type.
[heading Header]
// forwards to <boost/spirit/home/karma/auto.hpp>
#include <boost/spirit/include/karma_auto.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::auto_ // alias: boost::spirit::karma::auto_`]]
]
[heading Model of]
[:__primitive_generator_concept__]
[variablelist Notation
[[`s`] [A variable instance of any type for which a mapping to a
generator type is defined (the meta function
`traits::create_generator_exists` returns `mpl::true_`) or a
__karma_lazy_argument__ that evaluates to any type for which
a mapping to a generator type is defined (the meta function
`traits::create_generator_exists` returns `mpl::true_`).]]
]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __primitive_generator_concept__.
[table
[[Expression] [Description]]
[[`auto_`] [Create a generator instance compatible with the
supplied attribute type and use it for output
generation. This generator never fails
(unless the underlying output stream reports an
error).]]
[[`auto_(s)`] [Create a generator instance compatible with the
supplied literal value. This generator never fails
(unless the underlying output stream reports an
error).]]
]
[heading Additional Requirements]
The `auto_` generators can be used to emit output for any data type for which
a mapping to a generator type is defined (the meta function
`traits::create_generator_exists` returns `mpl::true_`).
The following table outlines the predefined mapping rules from the attribute type
to the generator type. These rules are applied recursively to create the generator
type which can be used to generate output from the given attribute type.
[table
[[Attribute type] [Generator type]]
[[`char`, `wchar_t`] [`standard::char_`, `standard_wide::char_`]]
[[`short`, `int`, `long`] [`short_`, `int_`, `long_`]]
[[`unsigned short`, `unsigned int`, `unsigned long`]
[`ushort_`, `uint_`, `ulong_`]]
[[`float`, `double`, `long double`] [`float_`, `double_`, `long_double`]]
[[`short`, `int`, `long`] [`short_`, `int_`, `long_`]]
[[`long long`, `unsigned long long`]
[`long_long`, `ulong_long`]]
[[`bool`] [`bool_`]]
[[Any string (`char const*`, `std::string`, etc.)]
[`string`]]
[[Any (STL) container] [Kleene Star (unary `'*'`)]]
[[Any Fusion sequence] [Sequence operator (`'<<'`)]]
[[`boost::optional<>`] [Optional operator (unary `'-'`)]]
[[`boost::variant<>`] [Alternative operator (`'|'`)]]
]
It is possible to add support for any custom data type by implementing a
specialization of the customization point __customize_create_generator__. This
customization can be used also to redefined any of the predefined mappings.
[heading Attributes]
[table
[[Expression] [Attribute]]
[[`auto_`] [`hold_any`, attribute is mandatory (otherwise
compilation will fail)]]
[[`auto_(s)`] [__unused__]]
]
[important The attribute type `hold_any` exposed by some of the `auto_`
generators is semantically and syntactically equivalent to
the type implemented by __boost_any__. It has been added to /Spirit/
as it has better a performance and a smaller footprint if compared to
__boost_any__.
]
[note In addition to their usual attribute of type `Attrib` all listed generators
accept an instance of a `boost::optional<Attrib>` as well. If the
`boost::optional<>` is initialized (holds a value) the generators behave
as if their attribute was an instance of `Attrib` and emit the value stored
in the `boost::optional<>`. Otherwise the generators will fail.]
[heading Complexity]
[:The complexity of the `auto_` generator depends on the attribute type. Each
attribute type results in a different generator type to be instantiated which
defines the overall complexity.]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_auto]
And a class definition used in the examples:
[reference_karma_complex]
[reference_karma_auto_complex]
Some usage examples of `auto_` generators:
[reference_karma_auto]
[endsect]

View File

@@ -0,0 +1,443 @@
[/==============================================================================
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2001-2011 Joel de Guzman
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:auxiliary Auxiliary Generators]
This module includes different auxiliary generators not fitting into any of the
other categories. It includes the `attr_cast`, `eol`, `eps`, and `lazy`
generators.
[heading Module Header]
// forwards to <boost/spirit/home/karma/auxiliary.hpp>
#include <boost/spirit/include/karma_auxiliary.hpp>
Also, see __include_structure__.
[/////////////////////////////////////////////////////////////////////////////]
[section:attr_cast Attribute Transformation Pseudo Generator (`attr_cast`)]
[heading Description]
The `attr_cast<Exposed, Transformed>()` component invokes the embedded generator
while supplying an attribute of type `Transformed`. The supplied attribute gets created
from the original attribute (of type `Exposed`) passed to this component using the
customization point __customize_transform_attribute__.
[heading Header]
// forwards to <boost/spirit/home/karma/auxiliary/attr_cast.hpp>
#include <boost/spirit/include/karma_attr_cast.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::attr_cast // alias: boost::spirit::karma::attr_cast`]]
]
[heading Synopsis]
template <Exposed, Transformed>
<unspecified> attr_cast(<unspecified>);
[heading Template parameters]
[table
[[Parameter] [Description] [Default]]
[[`Exposed`] [The type of the attribute supplied to the `attr_cast`.] [__unused_type__]]
[[`Transformed`][The type of the attribute expected by the embedded
generator `g`.] [__unused_type__]]
]
The `attr_cast` is a function template. It is possible to invoke it using the
following schemes:
attr_cast(g)
attr_cast<Exposed>(g)
attr_cast<Exposed, Transformed>(g)
depending on which of the attribute types can be deduced properly if not
explicitly specified.
[heading Model of]
[:__unary_generator_concept__]
[variablelist Notation
[[`g`] [A generator object.]]
]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __unary_generator_concept__.
[table
[[Expression] [Semantics]]
[[`attr_cast(g)`] [Create a component invoking the
generator `g` while passing an attribute of the type
as normally expected by `g`. The type of the supplied
attribute will be transformed to the type
`g` exposes as its attribute type (by using the
attribute customization point __customize_transform_attribute__).
This generator does not fail unless `g` fails.]]
[[`attr_cast<Exposed>(g)`] [Create a component invoking the
generator `g` while passing an attribute of the type
as normally expected by `g`. The supplied attribute
is expected to be of the type `Exposed`, it will be
transformed to the type `g` exposes as its attribute type
(using the attribute customization point
__customize_transform_attribute__).
This generator does not fail unless `g` fails.]]
[[`attr_cast<Exposed, Transformed>(g)`] [Create a component invoking the
generator `g` while passing an attribute of type
`Transformed`. The supplied attribute is expected
to be of the type `Exposed`, it will be transformed
to the type `Transformed` (using the attribute
customization point __customize_transform_attribute__).
This generator does not fail unless `g` fails.]]
]
[heading Attributes]
[table
[[Expression] [Attribute]]
[[`attr_cast(g)`] [`g: A --> attr_cast(g): A`]]
[[`attr_cast<Exposed>(g)`] [`g: A --> attr_cast<Exposed>(g): Exposed`]]
[[`attr_cast<Exposed, Transformed>(g)`]
[`g: A --> attr_cast<Exposed, Transformed>(g): Exposed`]]
]
[heading Complexity]
[:The complexity of this component is fully defined by the complexity of the
embedded generator `g`.]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_attr_cast]
The example references data structure `int_data` which needs a specialization of
the customization point __customize_transform_attribute__:
[reference_karma_auxiliary_attr_cast_data1]
Now we use the `attr_cast` pseudo generator to invoke the attribute
transformation:
[reference_karma_attr_cast1]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:eol End of Line Generator (`eol`)]
[heading Description]
The `eol` component generates a single newline character. It is equivalent
to `lit('\n')` or simply '\\n' (please see the [karma_char `char_`] generator
module for more details).
[heading Header]
// forwards to <boost/spirit/home/karma/auxiliary/eol.hpp>
#include <boost/spirit/include/karma_eol.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::eol // alias: boost::spirit::karma::eol`]]
]
[heading Model of]
[:__primitive_generator_concept__]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __primitive_generator_concept__.
[table
[[Expression] [Semantics]]
[[`eol`] [Create a component generating a single end of line
character in the output. This generator never fails
(unless the underlying output stream reports an
error).]]
]
[heading Attributes]
[table
[[Expression] [Attribute]]
[[`eol`] [__unused__]]
]
[heading Complexity]
[:O(1)]
The complexity is constant as a single character is generated in the output.
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_eol]
Basic usage of the `eol` generator:
[reference_karma_eol]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:eps Epsilon Generator (`eps`)]
The family of `eps` components allows to create pseudo generators generating
an empty string. This feature is sometimes useful either to force a generator
to fail or to succeed or to insert semantic actions into the generation process.
[heading Description]
The Epsilon (`eps`) is a multi-purpose generator that emits a zero length
string.
[heading Simple Form]
In its simplest form, `eps` creates a component generating an empty string
while always succeeding:
eps // always emits a zero-length string
This form is usually used to trigger a semantic action unconditionally.
For example, it is useful in triggering error messages when a set of
alternatives fail:
r = a | b | c | eps[error()]; // Call error if a, b, and c fail to generate
[heading Semantic Predicate]
The `eps(b)` component generates an empty string as well, but
succeeds only if `b` is `true` and fails otherwise. It's lazy variant `eps(fb)`
is equivalent to `eps(b)` except it evaluates the supplied function `fb` at
generate time, while using the return value as the criteria to succeed.
Semantic predicates allow you to attach a conditional function anywhere
in the grammar. In this role, the epsilon takes a __karma_lazy_argument__ that
returns `true` or `false`. The __karma_lazy_argument__ is typically a test
that is called to resolve ambiguity in the grammar. A generator failure will
be reported when the __karma_lazy_argument__ result evaluates to `false`.
Otherwise an empty string will be emitted. The general form is:
eps_p(fb) << rest;
The __karma_lazy_argument__ `fb` is called to do a semantic test. If the test
returns true, `rest` will be evaluated. Otherwise, the production will return
early without ever touching rest.
[heading Header]
// forwards to <boost/spirit/home/karma/auxiliary/eps.hpp>
#include <boost/spirit/include/karma_eps.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::eps // alias: boost::spirit::karma::eps`]]
]
[heading Model of]
[:__primitive_generator_concept__]
[variablelist Notation
[[`b`] [A boolean value.]]
[[`fb`] [A __karma_lazy_argument__ that evaluates to a boolean value.]]
]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __primitive_generator_concept__.
[table
[[Expression] [Semantics]]
[[`eps`] [Creates a component generating an empty string.
Succeeds always.]]
[[`eps(b)`] [Creates a component generating an empty string.
Succeeds if `b` is `true` (unless the underlying
output stream reports an error).]]
[[`eps(fb)`] [Creates a component generating an empty string.
Succeeds if `fb` returns `true` at generate time
(unless the underlying output stream reports an
error).]]
]
[heading Attributes]
[table
[[Expression] [Attribute]]
[[`eps`] [__unused__]]
[[`eps(b)`] [__unused__]]
[[`eps(fb)`] [__unused__]]
]
[heading Complexity]
[:O(1)]
The complexity is constant as no output is generated.
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_eps]
Basic usage of the `eps` generator:
[reference_karma_eps]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:lazy Lazy Generator (`lazy`)]
[heading Description]
The family of `lazy` components allows to use a dynamically returned generator
component for output generation. It calls the provided function or function
object at generate time using its return value as the actual generator to
produce the output.
[heading Header]
// forwards to <boost/spirit/home/karma/auxiliary/lazy.hpp>
#include <boost/spirit/include/karma_lazy.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::lazy // alias: boost::spirit::karma::lazy`]]
]
[heading Model of]
[:__generator_concept__]
[variablelist Notation
[[`fg`] [A function or function object that evaluates to a generator
object (an object exposing the __generator_concept__). This
function will be invoked at generate time.]]
]
The signature of `fg` is expected to be
G f(Unused, Context)
where `G`, the function's return value, is the type of the generator to be
invoked, and `Context` is the generator's __karma_context__ type (The
first argument is __unused__ to make the `Context` the second argument. This
is done for uniformity with __karma_actions__).
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __generator_concept__.
[table
[[Expression] [Semantics]]
[[`fg`] [The __phoenix__ function object `fg` will be
invoked at generate time. It is expected to return a
generator instance. This generator is then invoked
in order to generate the output. This generator will
succeed as long as the invoked generated succeeds as
well (unless the underlying output stream reports
an error).]]
[[`lazy(fg)`] [The function or function object will be invoked at
generate time. It is expected to return a generator
instance (note this version of `lazy` does not
require `fg` to be a __phoenix__ function
object). This generator is then invoked in order to
generate the output. This generator will succeed as
long as the invoked generated succeeds as well (except
if the underlying output stream reports an error).]]
]
[heading Attributes]
[table
[[Expression] [Attribute]]
[[`fg`] [The attribute type `G` as exposed by the generator `g`
returned from `fg`.]]
[[`lazy(fg)`] [The attribute type `G` as exposed by the generator `g`
returned from `fg`.]]
]
[heading Complexity]
The complexity of the `lazy` component is determined by the complexity of the
generator returned from `fg`.
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_lazy]
Basic usage of the `lazy` generator:
[reference_karma_lazy]
[endsect]
[endsect]

View File

@@ -0,0 +1,108 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:basics Generator Basics]
[heading Lazy Argument]
Some generators (e.g. primitives and non-terminals) may take in additional
attributes. Such generators take the form:
g(a1, a2,..., aN)
where `g` is a generator. Each of the arguments (a1 ... aN) can either be an
immediate value, or a function, `f`, with signature:
T f(Unused, Context)
where `T`, the function's return value, is compatible with the argument
type expected and `Context` is the generator's __karma_context__ type (The
first argument is __unused__ to make the `Context` the second argument. This
is done for uniformity with __karma_actions__).
[heading Character Encoding Namespace]
Some generators need to know which character set a `char` or `wchar_t` is
operating on. For example, the `alnum` generator works differently with
ISO8859.1 and ASCII encodings. Where necessary, Spirit encodes (tags)
the generator with the character set.
We have a namespace for each character set Spirit will be supporting.
That includes `ascii`, `iso8859_1`, `standard` and `standard_wide` (and
in the future, `unicode`). In each of the character encoding namespaces,
we place tagged versions of generators such as `alnum`, `space` etc.
Example:
using boost::spirit::ascii::space; // use the ASCII space generator
Namespaces:
* boost::spirit::ascii
* boost::spirit::iso8859_1
* boost::spirit::standard
* boost::spirit::standard_wide
For ease of use, the components in this namespaces are also brought into
the karma sub-namespaces with the same names:
* boost::spirit::karma::ascii
* boost::spirit::karma::iso8859_1
* boost::spirit::karma::standard
* boost::spirit::karma::standard_wide
[heading Examples]
All sections in the reference present some real world examples. The
examples use a common test harness to keep the example code as minimal
and direct to the point as possible. The test harness is presented
below.
Some includes:
[reference_karma_includes]
The used output iterator:
[reference_karma_output_iterator]
Our test functions:
This one tests the generators without attributes.
[reference_karma_test]
These test the generators with one or more user supplied attributes.
[reference_karma_test_attr]
[reference_karma_test_attr2]
This tests the generators with one attribute and while using delimited output.
[reference_karma_test_attr_delim]
The examples of the binary generators use one or more of the following tests.
[reference_karma_binary_test]
[reference_karma_binary_test_attr]
[heading Models]
Predefined models include:
* any literal string, e.g. "Hello, World",
* a pointer/reference to a null-terminated array of characters
* a `std::basic_string<Char>`
The namespace `boost::spirit::traits` is open for users to provide their
own specializations. The customization points implemented by __karma__ usable
to customize the behavior of generators are described in the section
__sec_customization_points__.
[endsect]

View File

@@ -0,0 +1,508 @@
[/==============================================================================
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2001-2011 Joel de Guzman
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:binary Binary Generators]
This module includes different generators allowing to output binary data.
It includes generators for default, little, and big endian binary output and
a `pad` generator allowing to control padding of the generated output
stream.
[heading Module Header]
// forwards to <boost/spirit/home/karma/binary.hpp>
#include <boost/spirit/include/karma_binary.hpp>
Also, see __include_structure__.
[/////////////////////////////////////////////////////////////////////////////]
[section:binary_native Binary Native Endianness Generators]
[heading Description]
The binary native endianness generators described in this section are used to
emit binary byte streams laid out conforming to the native endianness (byte
order) of the target architecture.
[heading Header]
// forwards to <boost/spirit/home/karma/binary.hpp>
#include <boost/spirit/include/karma_binary.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::byte_ // alias: boost::spirit::karma::byte_` ]]
[[`boost::spirit::word // alias: boost::spirit::karma::word` ]]
[[`boost::spirit::dword // alias: boost::spirit::karma::dword` ]]
[[`boost::spirit::qword // alias: boost::spirit::karma::qword` ]]
[[`boost::spirit::bin_float // alias: boost::spirit::karma::bin_float` ]]
[[`boost::spirit::bin_double // alias: boost::spirit::karma::bin_double` ]]
]
[note The generators `qword` and `qword(qw)` are only available on
platforms where the preprocessor constant `BOOST_HAS_LONG_LONG` is
defined (i.e. on platforms having native support for `unsigned long long`
(64 bit) integer types).]
[heading Model of]
[:__primitive_generator_concept__]
[variablelist Notation
[[`b`] [A single byte (8 bit binary value) or a __karma_lazy_argument__
that evaluates to a single byte]]
[[`w`] [A 16 bit binary value or a __karma_lazy_argument__ that
evaluates to a 16 bit binary value. This value is always
interpreted using native endianness.]]
[[`dw`] [A 32 bit binary value or a __karma_lazy_argument__ that
evaluates to a 32 bit binary value. This value is always
interpreted using native endianness.]]
[[`qw`] [A 64 bit binary value or a __karma_lazy_argument__ that
evaluates to a 64 bit binary value. This value is always
interpreted using native endianness.]]
[[`f`] [A float binary value or a __karma_lazy_argument__ that
evaluates to a float binary value. This value is always
interpreted using native endianness.]]
[[`d`] [A double binary value or a __karma_lazy_argument__ that
evaluates to a double binary value. This value is always
interpreted using native endianness.]]
]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __primitive_generator_concept__.
[table
[[Expression] [Description]]
[[`byte_`] [Output the binary representation of the least
significant byte of the mandatory attribute. This
generator never fails (unless the underlying
output stream reports an error).]]
[[`word`] [Output the binary representation of the least
significant 16 bits of the mandatory attribute
in native endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`dword`] [Output the binary representation of the least
significant 32 bits of the mandatory attribute
in native endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`qword`] [Output the binary representation of the least
significant 64 bits of the mandatory attribute
in native endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`bin_float`] [Output the binary representation of the mandatory
float attribute in native endian representation.
This generator never fails (unless the underlying
output stream reports an error).]]
[[`bin_double`] [Output the binary representation of the mandatory
double attribute in native endian representation.
This generator never fails (unless the underlying
output stream reports an error).]]
[[`byte_(b)`] [Output the binary representation of the least
significant byte of the immediate parameter. This
generator never fails (unless the underlying
output stream reports an error).]]
[[`word(w)`] [Output the binary representation of the least
significant 16 bits of the immediate parameter
in native endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`dword(dw)`] [Output the binary representation of the least
significant 32 bits of the immediate parameter
in native endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`qword(qw)`] [Output the binary representation of the least
significant 64 bits of the immediate parameter
in native endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`bin_float(f)`] [Output the binary representation of the immediate
float parameter in native endian representation.
This generator never fails (unless the underlying
output stream reports an error).]]
[[`bin_double(d)`] [Output the binary representation of the immediate
double parameter in native endian representation.
This generator never fails (unless the underlying
output stream reports an error).]]
]
[heading Attributes]
[table
[[Expression] [Attribute]]
[[`byte_`] [`boost::uint_least8_t`, attribute is mandatory
(otherwise compilation will fail)]]
[[`word`] [`boost::uint_least16_t`, attribute is mandatory
(otherwise compilation will fail)]]
[[`dword`] [`boost::uint_least32_t`, attribute is mandatory
(otherwise compilation will fail)]]
[[`qword`] [`boost::uint_least64_t`, attribute is mandatory
(otherwise compilation will fail)]]
[[`bin_float`] [`float`, attribute is mandatory
(otherwise compilation will fail)]]
[[`bin_double`] [`double`, attribute is mandatory
(otherwise compilation will fail)]]
[[`byte_(b)`] [__unused__]]
[[`word(w)`] [__unused__]]
[[`dword(dw)`] [__unused__]]
[[`qword(qw)`] [__unused__]]
[[`bin_float(f)`] [__unused__]]
[[`bin_double(d)`] [__unused__]]
]
[note In addition to their usual attribute of type `Attrib` all listed generators
accept an instance of a `boost::optional<Attrib>` as well. If the
`boost::optional<>` is initialized (holds a value) the generators behave
as if their attribute was an instance of `Attrib` and emit the value stored
in the `boost::optional<>`. Otherwise the generators will fail.]
[heading Complexity]
[:O(N), where N is the number of bytes emitted by the binary generator]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_native_binary]
Basic usage of the native binary generators with some results for little endian
platforms:
[reference_karma_native_binary_little]
Basic usage of the native binary generators with some results for big endian
platforms:
[reference_karma_native_binary_big]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:binary_little Binary Little Endianness Generators]
[heading Description]
The little native endianness generators described in this section are used to
emit binary byte streams laid out conforming to the little endianness byte
order.
[heading Header]
// forwards to <boost/spirit/home/karma/binary.hpp>
#include <boost/spirit/include/karma_binary.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::little_word // alias: boost::spirit::karma::little_word` ]]
[[`boost::spirit::little_dword // alias: boost::spirit::karma::little_dword` ]]
[[`boost::spirit::little_qword // alias: boost::spirit::karma::little_qword` ]]
[[`boost::spirit::little_bin_float // alias: boost::spirit::karma::little_bin_float` ]]
[[`boost::spirit::little_bin_double // alias: boost::spirit::karma::little_bin_double` ]]
]
[note The generators `little_qword` and `little_qword(qw)` are only available on
platforms where the preprocessor constant `BOOST_HAS_LONG_LONG` is
defined (i.e. on platforms having native support for `unsigned long long`
(64 bit) integer types).]
[heading Model of]
[:__primitive_generator_concept__]
[variablelist Notation
[[`w`] [A 16 bit binary value or a __karma_lazy_argument__ that
evaluates to a 16 bit binary value. This value is always
interpreted using native endianness.]]
[[`dw`] [A 32 bit binary value or a __karma_lazy_argument__ that
evaluates to a 32 bit binary value. This value is always
interpreted using native endianness.]]
[[`qw`] [A 64 bit binary value or a __karma_lazy_argument__ that
evaluates to a 64 bit binary value. This value is always
interpreted using native endianness.]]
[[`f`] [A float binary value or a __karma_lazy_argument__ that
evaluates to a float binary value. This value is always
interpreted using native endianness.]]
[[`d`] [A double binary value or a __karma_lazy_argument__ that
evaluates to a double binary value. This value is always
interpreted using native endianness.]]
]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __primitive_generator_concept__.
[table
[[Expression] [Description]]
[[`little_word`] [Output the binary representation of the least
significant 16 bits of the mandatory attribute
in little endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`little_dword`] [Output the binary representation of the least
significant 32 bits of the mandatory attribute
in little endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`little_qword`] [Output the binary representation of the least
significant 64 bits of the mandatory attribute
in little endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`little_bin_float`] [Output the binary representation of the mandatory
float attribute in little endian representation.
This generator never fails (unless the underlying
output stream reports an error).]]
[[`little_bin_double`] [Output the binary representation of the mandatory
double attribute in little endian representation.
This generator never fails (unless the underlying
output stream reports an error).]]
[[`little_word(w)`] [Output the binary representation of the least
significant 16 bits of the immediate parameter
in little endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`little_dword(dw)`] [Output the binary representation of the least
significant 32 bits of the immediate parameter
in little endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`little_qword(qw)`] [Output the binary representation of the least
significant 64 bits of the immediate parameter
in little endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`little_bin_float(f)`] [Output the binary representation of the immediate
float parameter in little endian representation.
This generator never fails (unless the underlying
output stream reports an error).]]
[[`little_bin_double(d)`] [Output the binary representation of the immediate
double parameter in little endian representation.
This generator never fails (unless the underlying
output stream reports an error).]]
]
[heading Attributes]
[table
[[Expression] [Attribute]]
[[`little_word`] [`boost::uint_least16_t`, attribute is mandatory
(otherwise compilation will fail)]]
[[`little_dword`] [`boost::uint_least32_t`, attribute is mandatory
(otherwise compilation will fail)]]
[[`little_qword`] [`boost::uint_least64_t`, attribute is mandatory
(otherwise compilation will fail)]]
[[`little_bin_float`] [`float`, attribute is mandatory
(otherwise compilation will fail)]]
[[`little_bin_double`] [`double`, attribute is mandatory
(otherwise compilation will fail)]]
[[`little_word(w)`] [__unused__]]
[[`little_dword(dw)`] [__unused__]]
[[`little_qword(qw)`] [__unused__]]
[[`little_bin_float(f)`] [__unused__]]
[[`little_bin_double(d)`] [__unused__]]
]
[heading Complexity]
[:O(N), where N is the number of bytes emitted by the binary generator]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_little_binary]
Basic usage of the little binary generators:
[reference_karma_little_binary]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:binary_big Binary Big Endianness Generators]
[heading Description]
The big native endianness generators described in this section are used to
emit binary byte streams laid out conforming to the big endianness byte
order.
[heading Header]
// forwards to <boost/spirit/home/karma/binary.hpp>
#include <boost/spirit/include/karma_binary.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::big_word // alias: boost::spirit::karma::big_word` ]]
[[`boost::spirit::big_dword // alias: boost::spirit::karma::big_dword` ]]
[[`boost::spirit::big_qword // alias: boost::spirit::karma::big_qword` ]]
[[`boost::spirit::big_bin_float // alias: boost::spirit::karma::big_bin_float` ]]
[[`boost::spirit::big_bin_double // alias: boost::spirit::karma::big_bin_double` ]]
]
[note The generators `big_qword` and `big_qword(qw)` are only available on
platforms where the preprocessor constant `BOOST_HAS_LONG_LONG` is
defined (i.e. on platforms having native support for `unsigned long long`
(64 bit) integer types).]
[heading Model of]
[:__primitive_generator_concept__]
[variablelist Notation
[[`w`] [A 16 bit binary value or a __karma_lazy_argument__ that
evaluates to a 16 bit binary value. This value is always
interpreted using native endianness.]]
[[`dw`] [A 32 bit binary value or a __karma_lazy_argument__ that
evaluates to a 32 bit binary value. This value is always
interpreted using native endianness.]]
[[`qw`] [A 64 bit binary value or a __karma_lazy_argument__ that
evaluates to a 64 bit binary value. This value is always
interpreted using native endianness.]]
[[`f`] [A float binary value or a __karma_lazy_argument__ that
evaluates to a float binary value. This value is always
interpreted using native endianness.]]
[[`d`] [A double binary value or a __karma_lazy_argument__ that
evaluates to a double binary value. This value is always
interpreted using native endianness.]]
]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __primitive_generator_concept__.
[table
[[Expression] [Description]]
[[`big_word`] [Output the binary representation of the least
significant 16 bits of the mandatory attribute
in big endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`big_dword`] [Output the binary representation of the least
significant 32 bits of the mandatory attribute
in big endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`big_qword`] [Output the binary representation of the least
significant 64 bits of the mandatory attribute
in big endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`big_bin_float`] [Output the binary representation of the mandatory
float attribute in big endian representation.
This generator never fails (unless the underlying
output stream reports an error).]]
[[`big_bin_double`] [Output the binary representation of the mandatory
double attribute in big endian representation.
This generator never fails (unless the underlying
output stream reports an error).]]
[[`big_word(w)`] [Output the binary representation of the least
significant 16 bits of the immediate parameter
in big endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`big_dword(dw)`] [Output the binary representation of the least
significant 32 bits of the immediate parameter
in big endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`big_qword(qw)`] [Output the binary representation of the least
significant 64 bits of the immediate parameter
in big endian representation. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`big_bin_float(f)`] [Output the binary representation of the immediate
float parameter in big endian representation.
This generator never fails (unless the underlying
output stream reports an error).]]
[[`big_bin_double(d)`] [Output the binary representation of the immediate
double parameter in big endian representation.
This generator never fails (unless the underlying
output stream reports an error).]]
]
[heading Attributes]
[table
[[Expression] [Attribute]]
[[`big_word`] [`boost::uint_least16_t`, attribute is mandatory
(otherwise compilation will fail)]]
[[`big_dword`] [`boost::uint_least32_t`, attribute is mandatory
(otherwise compilation will fail)]]
[[`big_qword`] [`boost::uint_least64_t`, attribute is mandatory
(otherwise compilation will fail)]]
[[`big_bin_float`] [`float`, attribute is mandatory
(otherwise compilation will fail)]]
[[`big_bin_double`] [`double`, attribute is mandatory
(otherwise compilation will fail)]]
[[`big_word(w)`] [__unused__]]
[[`big_dword(dw)`] [__unused__]]
[[`big_qword(qw)`] [__unused__]]
[[`big_bin_float(f)`] [__unused__]]
[[`big_bin_double(d)`] [__unused__]]
]
[heading Complexity]
[:O(N), where N is the number of bytes emitted by the binary generator]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_big_binary]
Basic usage of the big binary generators:
[reference_karma_big_binary]
[endsect]
[endsect]

View File

@@ -0,0 +1,491 @@
[/==============================================================================
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2001-2011 Joel de Guzman
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:char Char Generators]
This module includes different character oriented generators allowing to output
single characters. Currently, it includes literal chars (e.g. `'x'`, `L'x'`),
`char_` (single characters, ranges and character sets) and the encoding
specific character classifiers (`alnum`, `alpha`, `digit`, `xdigit`, etc.).
[heading Module Header]
// forwards to <boost/spirit/home/karma/char.hpp>
#include <boost/spirit/include/karma_char.hpp>
Also, see __include_structure__.
[/////////////////////////////////////////////////////////////////////////////]
[section:char_generator Character Generators (`char_`, `lit`)]
[heading Description]
The character generators described in this section are:
The `char_` generator emits single characters. The `char_` generator has an
associated __karma_char_encoding_namespace__. This is needed when doing basic
operations such as forcing lower or upper case and dealing with
character ranges.
There are various forms of `char_`.
[heading char_]
The no argument form of `char_` emits any character in the associated
__karma_char_encoding_namespace__.
char_ // emits any character as supplied by the attribute
[heading char_(ch)]
The single argument form of `char_` (with a character argument) emits
the supplied character.
char_('x') // emits 'x'
char_(L'x') // emits L'x'
char_(x) // emits x (a char)
[heading char_(first, last)]
`char_` with two arguments, emits any character from a range of characters as
supplied by the attribute.
char_('a','z') // alphabetic characters
char_(L'0',L'9') // digits
A range of characters is created from a low-high character pair. Such a
generator emits a single character that is in the range, including both
endpoints. Note, the first character must be /before/ the second,
according to the underlying __karma_char_encoding_namespace__.
Character mapping is inherently platform dependent. It is not guaranteed
in the standard for example that `'A' < 'Z'`, that is why in Spirit2, we
purposely attach a specific __karma_char_encoding_namespace__ (such as ASCII,
ISO-8859-1) to the `char_` generator to eliminate such ambiguities.
[note *Sparse bit vectors*
To accommodate 16/32 and 64 bit characters, the char-set statically
switches from a `std::bitset` implementation when the character type is
not greater than 8 bits, to a sparse bit/boolean set which uses a sorted
vector of disjoint ranges (`range_run`). The set is constructed from
ranges such that adjacent or overlapping ranges are coalesced.
`range_runs` are very space-economical in situations where there are lots
of ranges and a few individual disjoint values. Searching is O(log n)
where n is the number of ranges.]
[heading char_(def)]
Lastly, when given a string (a plain C string, a `std::basic_string`,
etc.), the string is regarded as a char-set definition string following
a syntax that resembles posix style regular expression character sets
(except that double quotes delimit the set elements instead of square
brackets and there is no special negation ^ character). Examples:
char_("a-zA-Z") // alphabetic characters
char_("0-9a-fA-F") // hexadecimal characters
char_("actgACTG") // DNA identifiers
char_("\x7f\x7e") // Hexadecimal 0x7F and 0x7E
These generators emit any character from a range of characters as
supplied by the attribute.
[heading lit(ch)]
`lit`, when passed a single character, behaves like the single argument
`char_` except that `lit` does not consume an attribute. A plain
`char` or `wchar_t` is equivalent to a `lit`.
[note `lit` is reused by the [karma_string String Generators], the
char generators, and the Numeric Generators (see [signed_int signed integer],
[unsigned_int unsigned integer], and [real_number real number] generators). In
general, a char generator is created when you pass in a
character, a string generator is created when you pass in a string, and a
numeric generator is created when you use a numeric literal. The
exception is when you pass a single element literal string, e.g.
`lit("x")`. In this case, we optimize this to create a char generator
instead of a string generator.]
Examples:
'x'
lit('x')
lit(L'x')
lit(c) // c is a char
[heading Header]
// forwards to <boost/spirit/home/karma/char/char.hpp>
#include <boost/spirit/include/karma_char_.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::lit // alias: boost::spirit::karma::lit` ]]
[[`ns::char_`]]
]
In the table above, `ns` represents a __karma_char_encoding_namespace__.
[heading Model of]
[:__primitive_generator_concept__]
[variablelist Notation
[[`ch`, `ch1`, `ch2`]
[Character-class specific character (See __char_class_types__),
or a __karma_lazy_argument__ that evaluates to a
character-class specific character value]]
[[`cs`] [Character-set specifier string (See
__char_class_types__), or a __karma_lazy_argument__ that
evaluates to a character-set specifier string, or a
pointer/reference to a null-terminated array of characters.
This string specifies a char-set definition string following
a syntax that resembles posix style regular expression character
sets (except the square brackets and the negation `^` character).]]
[[`ns`] [A __karma_char_encoding_namespace__.]]
[[`cg`] [A char generator, a char range generator, or a char set generator.]]]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __primitive_generator_concept__.
[table
[[Expression] [Description]]
[[`ch`] [Generate the character literal `ch`. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`lit(ch)`] [Generate the character literal `ch`. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`ns::char_`] [Generate the character provided by a mandatory
attribute interpreted in the character set defined
by `ns`. This generator never fails (unless the
underlying output stream reports an error).]]
[[`ns::char_(ch)`] [Generate the character `ch` as provided by the
immediate literal value the generator is initialized
from. If this generator has an associated attribute
it succeeds only as long as the attribute is equal
to the immediate literal (unless the underlying
output stream reports an error). Otherwise this
generator fails and does not generate any output.]]
[[`ns::char_("c")`] [Generate the character `c` as provided by the
immediate literal value the generator is initialized
from. If this generator has an associated attribute
it succeeds only as long as the attribute is equal
to the immediate literal (unless the underlying
output stream reports an error). Otherwise this
generator fails and does not generate any output.]]
[[`ns::char_(ch1, ch2)`][Generate the character provided by a mandatory
attribute interpreted in the character set defined
by `ns`. The generator succeeds as long as the
attribute belongs to the character range `[ch1, ch2]`
(unless the underlying output stream reports an
error). Otherwise this generator fails and does not
generate any output.]]
[[`ns::char_(cs)`] [Generate the character provided by a mandatory
attribute interpreted in the character set defined
by `ns`. The generator succeeds as long as the
attribute belongs to the character set `cs`
(unless the underlying output stream reports an
error). Otherwise this generator fails and does not
generate any output.]]
[[`~cg`] [Negate `cg`. The result is a negated char generator
that inverts the test condition of the character
generator it is attached to.]]
]
A character `ch` is assumed to belong to the character range defined by
`ns::char_(ch1, ch2)` if its character value (binary representation)
interpreted in the character set defined by `ns` is not smaller than the
character value of `ch1` and not larger then the character value of `ch2` (i.e.
`ch1 <= ch <= ch2`).
The `charset` parameter passed to `ns::char_(charset)` must be a string
containing more than one character. Every single character in this string is
assumed to belong to the character set defined by this expression. An exception
to this is the `'-'` character which has a special meaning if it is not
specified as the first and not the last character in `charset`. If the `'-'`
is used in between to characters it is interpreted as spanning a character
range. A character `ch` is considered to belong to the defined character set
`charset` if it matches one of the characters as specified by the string
parameter described above. For example
[table
[[Example] [Description]]
[[`char_("abc")`] ['a', 'b', and 'c']]
[[`char_("a-z")`] [all characters (and including) from 'a' to 'z']]
[[`char_("a-zA-Z")`] [all characters (and including) from 'a' to 'z' and 'A' and 'Z']]
[[`char_("-1-9")`] ['-' and all characters (and including) from '1' to '9']]
]
[heading Attributes]
[table
[[Expression] [Attribute]]
[[`ch`] [__unused__]]
[[`lit(ch)`] [__unused__]]
[[`ns::char_`] [`Ch`, attribute is mandatory (otherwise compilation
will fail). `Ch` is the character type of the
__karma_char_encoding_namespace__, `ns`.]]
[[`ns::char_(ch)`] [`Ch`, attribute is optional, if it is supplied, the
generator compares the attribute with `ch` and
succeeds only if both are equal, failing otherwise.
`Ch` is the character type of the
__karma_char_encoding_namespace__, `ns`.]]
[[`ns::char_("c")`] [`Ch`, attribute is optional, if it is supplied, the
generator compares the attribute with `c` and
succeeds only if both are equal, failing otherwise.
`Ch` is the character type of the
__karma_char_encoding_namespace__, `ns`.]]
[[`ns::char_(ch1, ch2)`][`Ch`, attribute is mandatory (otherwise compilation
will fail), the generator succeeds if the attribute
belongs to the character range `[ch1, ch2]`
interpreted in the character set defined by `ns`.
`Ch` is the character type of the
__karma_char_encoding_namespace__, `ns`.]]
[[`ns::char_(cs)`] [`Ch`, attribute is mandatory (otherwise compilation
will fail), the generator succeeds if the attribute
belongs to the character set `cs`, interpreted
in the character set defined by `ns`.
`Ch` is the character type of the
__karma_char_encoding_namespace__, `ns`.]]
[[`~cg`] [Attribute of `cg`]]
]
[note In addition to their usual attribute of type `Ch` all listed generators
accept an instance of a `boost::optional<Ch>` as well. If the
`boost::optional<>` is initialized (holds a value) the generators behave
as if their attribute was an instance of `Ch` and emit the value stored
in the `boost::optional<>`. Otherwise the generators will fail.]
[heading Complexity]
[:O(1)]
The complexity of `ch`, `lit(ch)`, `ns::char_`, `ns::char_(ch)`, and
`ns::char_("c")` is constant as all generators emit exactly one character per
invocation.
The character range generator (`ns::char_(ch1, ch2)`) additionally requires
constant lookup time for the verification whether the attribute belongs to
the character range.
The character set generator (`ns::char_(cs)`) additionally requires
O(log N) lookup time for the verification whether the attribute belongs to
the character set, where N is the number of characters in the character set.
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_char]
Basic usage of `char_` generators:
[reference_karma_char]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:char_class Character Classification Generators (`alnum`, `digit`, etc.)]
[heading Description]
The library has the full repertoire of single character generators for
character classification. This includes the usual `alnum`, `alpha`,
`digit`, `xdigit`, etc. generators. These generators have an associated
__karma_char_encoding_namespace__. This is needed when doing basic operations
such as forcing lower or upper case.
[heading Header]
// forwards to <boost/spirit/home/karma/char/char_class.hpp>
#include <boost/spirit/include/karma_char_class.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`ns::alnum`]]
[[`ns::alpha`]]
[[`ns::blank`]]
[[`ns::cntrl`]]
[[`ns::digit`]]
[[`ns::graph`]]
[[`ns::lower`]]
[[`ns::print`]]
[[`ns::punct`]]
[[`ns::space`]]
[[`ns::upper`]]
[[`ns::xdigit`]]
]
In the table above, `ns` represents a __karma_char_encoding_namespace__ used by the
corresponding character class generator. All listed generators have a mandatory
attribute `Ch` and will not compile if no attribute is associated.
[heading Model of]
[:__primitive_generator_concept__]
[variablelist Notation
[[`ns`] [A __karma_char_encoding_namespace__.]]]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __primitive_generator_concept__.
[table
[[Expression] [Semantics]]
[[`ns::alnum`] [If the mandatory attribute satisfies the concept of
`std::isalnum` in the __karma_char_encoding_namespace__
the generator succeeds after emitting
its attribute (unless the underlying output stream
reports an error). This generator fails otherwise
while not generating anything.]]
[[`ns::alpha`] [If the mandatory attribute satisfies the concept of
`std::isalpha` in the __karma_char_encoding_namespace__
the generator succeeds after emitting
its attribute (unless the underlying output stream
reports an error). This generator fails otherwise
while not generating anything.]]
[[`ns::blank`] [If the mandatory attribute satisfies the concept of
`std::isblank` in the __karma_char_encoding_namespace__
the generator succeeds after emitting
its attribute (unless the underlying output stream
reports an error). This generator fails otherwise
while not generating anything.]]
[[`ns::cntrl`] [If the mandatory attribute satisfies the concept of
`std::iscntrl` in the __karma_char_encoding_namespace__
the generator succeeds after emitting
its attribute (unless the underlying output stream
reports an error). This generator fails otherwise
while not generating anything.]]
[[`ns::digit`] [If the mandatory attribute satisfies the concept of
`std::isdigit` in the __karma_char_encoding_namespace__
the generator succeeds after emitting
its attribute (unless the underlying output stream
reports an error). This generator fails otherwise
while not generating anything.]]
[[`ns::graph`] [If the mandatory attribute satisfies the concept of
`std::isgraph` in the __karma_char_encoding_namespace__
the generator succeeds after emitting
its attribute (unless the underlying output stream
reports an error). This generator fails otherwise
while not generating anything.]]
[[`ns::print`] [If the mandatory attribute satisfies the concept of
`std::isprint` in the __karma_char_encoding_namespace__
the generator succeeds after emitting
its attribute (unless the underlying output stream
reports an error). This generator fails otherwise
while not generating anything.]]
[[`ns::punct`] [If the mandatory attribute satisfies the concept of
`std::ispunct` in the __karma_char_encoding_namespace__
the generator succeeds after emitting
its attribute (unless the underlying output stream
reports an error). This generator fails otherwise
while not generating anything.]]
[[`ns::xdigit`] [If the mandatory attribute satisfies the concept of
`std::isxdigit` in the __karma_char_encoding_namespace__
the generator succeeds after emitting
its attribute (unless the underlying output stream
reports an error). This generator fails otherwise
while not generating anything.]]
[[`ns::lower`] [If the mandatory attribute satisfies the concept of
`std::islower` in the __karma_char_encoding_namespace__
the generator succeeds after emitting
its attribute (unless the underlying output stream
reports an error). This generator fails otherwise
while not generating anything.]]
[[`ns::upper`] [If the mandatory attribute satisfies the concept of
`std::isupper` in the __karma_char_encoding_namespace__
the generator succeeds after emitting
its attribute (unless the underlying output stream
reports an error). This generator fails otherwise
while not generating anything.]]
[[`ns::space`] [If the optional attribute satisfies the concept of
`std::isspace` in the __karma_char_encoding_namespace__
the generator succeeds after emitting
its attribute (unless the underlying output stream
reports an error). This generator fails otherwise
while not generating anything.If no attribute is
supplied this generator emits a single space
character in the character set defined by `ns`.]]
]
Possible values for `ns` are described in the section __karma_char_encoding_namespace__.
[note The generators `alpha` and `alnum` might seem to behave unexpected if
used inside a `lower[]` or `upper[]` directive. Both directives
additionally apply the semantics of `std::islower` or `std::isupper`
to the respective character class. Some examples:
``
std::string s;
std::back_insert_iterator<std::string> out(s);
generate(out, lower[alpha], 'a'); // succeeds emitting 'a'
generate(out, lower[alpha], 'A'); // fails
``
The generator directive `upper[]` behaves correspondingly.
]
[heading Attributes]
[:All listed character class generators can take any attribute `Ch`. All
character class generators (except `space`) require an attribute and will
fail compiling otherwise.]
[note In addition to their usual attribute of type `Ch` all listed generators
accept an instance of a `boost::optional<Ch>` as well. If the
`boost::optional<>` is initialized (holds a value) the generators behave
as if their attribute was an instance of `Ch` and emit the value stored
in the `boost::optional<>`. Otherwise the generators will fail.]
[heading Complexity]
[:O(1)]
The complexity is constant as the generators emit not more than one character
per invocation.
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_char_class]
Basic usage of an `alpha` generator:
[reference_karma_char_class]
[endsect]
[endsect]

View File

@@ -0,0 +1,298 @@
[/==============================================================================
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2001-2011 Joel de Guzman
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[/////////////////////////////////////////////////////////////////////////////]
[section:karma_complex Complex - A first more complex generator]
In this section we will develop a generator for complex numbers, allowing to
represent a `std::complex` either as `(real, imag)` (where `real` and `imag`
are the real and imaginary parts of the complex number) or as a simple `real`
if the imaginary part happens to be equal to zero. This example will highlight
the power of __karma__ allowing to combine compile time definition of
formatting rules with runtime based decisions which of the rules to apply.
Also this time, we're using __phoenix__ to do the semantic actions.
Our goal is to allow for two different output formats to be applied depending
on whether the imaginary part of the complex number is zero or not. Let's write
both as a set of alternatives:
'(' << double_ << ", " << double_ << ')'
| double_
where the first alternative should be used for numbers having a non-zero
imaginary part, while the second is for real numbers. Generally, alternatives
are tried in the sequence of their definition as long until one of the
expressions (as delimited by `'|'`) succeeds. If no generator expression
succeeds the whole alternative fails.
If we left this formatting grammar as is our generator would always choose
the first alternative. We need to add some additional rules allowing to make the
first alternative fail. So, if the first alternative fails the second one will
be chosen instead. The decision about whether to choose the first alternative
has to be made at runtime as only then we actually know the value of the
imaginary part of the complex number. __karma__ provides us with with a
primitive generator `eps()`, which is usable as a semantic predicate. It has
the property to 'succeed' generating only if its argument is true (while it
never generates any output on its own).
double imag = ...; // imaginary part
eps(imag != 0) << '(' << double_ << ", " << double_ << ')'
| double_
If one of the generator elements of a sequence fails the whole sequence will
fail. This is exactly what we need, forcing the second alternative to be chosen
for complex numbers with imaginary parts equal to zero.
[import ../../example/karma/complex_number.cpp]
Now on to the full example, this time with the proper semantic actions (the
complete cpp file for this example can be found here:
[@../../example/karma/complex_number.cpp complex_number.cpp]).
We will use the `std::complex` type for this and all subsequent related
examples. And here you can see the full code of the generator allowing to
output a complex number either as a pair of numbers (if the imaginary part is
non-zero) or as a single number (if the complex is a real number):
[tutorial_karma_complex_number]
The `double_` generators have this semantic action attached:
_1 = n
which passes `n` to the first element of the s generator's attached
semantic action. Remember, semantic actions in __karma__ are called
before the corresponding generator is invoked and they are expected
to provide the generator with the data to be used. The semantic action
above assigns the value to be generated (`n`) to the generator (actually,
the attribute of `double_`). `_1` is a Phoenix placeholder referring to
the attribute of the semantic action's attached generator. If you need
more information about semantic actions, you may want to read about them
in this section: __karma_actions__.
These semantic actions are easy to understand but have the unexpected side
effect of being slightly less efficient than it could be. In addition they tend
to make the formatting grammar less readable. We will see in one of the next
sections how it is possible to use other, built-in features of __karma__ to get
rid of the semantic actions altogether. When writing your grammars in Spirit
you should always try to avoid semantic actions which is often possible.
Semantic actions are really powerful tools but grammars tend to be more
efficient and readable without them.
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:karma_easier_complex Complex - Made easier]
[import ../../example/karma/complex_number_easier.cpp]
In the previous section we showed how to format a complex number (i.e.
a pair of doubles). In this section we will build on this example with the goal
to avoid using semantic actions in the format specification. Let's have a look
at the resulting code first, trying to understand it afterwards (the full source
file for this example can be found here:
[@../../example/karma/complex_number_easier.cpp complex_number_easier.cpp]):
[tutorial_karma_complex_number_easier]
Let's cover some basic library features first.
[heading Making Numeric Generators Fail]
All __karma_numeric__ (such as `double_`, et.al.) take the value to
emit from an attached attribute.
double d = 1.5;
generate(out, double_, d); // will emit '1.5' (without the quotes)
Alternatively, they may be initialized from a literal value. For instance, to
emit a constant `1.5` you may write:
generate(out, double_(1.5)); // will emit '1.5' as well (without the quotes)
The difference to a simple `1.5` or `lit(1.5)` is that the `double_(1.5)`
consumes an attribute if one is available. Additionally, it compares its
immediate value to the value of the supplied attribute, and fails if those are
not equal.
double d = 1.5;
generate(out, double_(1.5), d); // will emit '1.5' as long as d == 1.5
This feature, namely to succeed generating only if the attribute matches the
immediate value, enables numeric generators to be used to dynamically control
the way output is generated.
[note Quite a few generators will fail if their immediate value is not equal
to the supplied attribute. Among those are all __karma_char__ and
all [karma_string String Generators]. Generally,
all generators having a sibling created by a variant of `lit()` belong
into this category.]
[heading Predicates - The Conditionals for Output Generators]
In addition to the __karma_eps__ generator mentioned earlier __karma__ provides
two special operators enabling dynamic flow control: the
__karma_and_predicate__ and the __karma_not_predicate__. The main property of
both predicates is to discard all output emitted by the attached generator.
This is equivalent to the behavior of predicates used for
parsing. There the predicates do not consume any input allowing to look ahead
in the input stream. In Karma, the and predicate succeeds as long as its
associated generator succeeds, while the not predicate succeeds only if its
associated generator fails.
[note The generator predicates in __karma__ consume an attribute, if
available. This makes them behave differently from predicates in __qi__,
where they do not expose any attribute. This is because predicates
allow to make decisions based on data available only at runtime. While
in __qi__ during parsing the decision is made based on looking ahead
a few more input tokens, in __karma__ the criteria has to be supplied
by the user. The simplest way to do this is by providing an attribute.]
As an example, the following generator succeeds generating
double d = 1.0;
BOOST_ASSERT(generate(out, &double_(1.0), d)); // succeeds as d == 1.0
while this one will fail:
double d = 1.0;
BOOST_ASSERT(!generate(out, !double_(1.0), d)); // fails as d == 1.0
Neither of these will emit any output. The predicates discard everything
emitted by the generators to which they are applied.
[heading Ignoring Supplied Attributes]
Sometimes it is desirable to 'skip' (i.e. ignore) a provided attribute. This
happens for instance in alternative generators, where some of the alternatives
need to extract only part of the overall attribute passed to the alternative
generator. __karma__ has a special pseudo generator for that: the directive
__karma_omit__`[]`. This directive consumes an attribute of the type defined by its
embedded generator but it does not emit any output.
[note The __karma__ __karma_omit__ directive does the 'opposite' of the
directive of the same name in __qi__. While the __qi_omit__ in __qi__
consumes input without exposing an attribute, its __karma__ counterpart
consumes an attribute without emitting any output.
]
[heading Putting everything together]
Very similar to our first example earlier we use two alternatives to allow for
the two different output formats depending on whether the imaginary part of the
complex number is equal to zero or not. The first alternative is executed if the
imaginary part is not zero, the second alternative otherwise. This time we make
the decision during runtime using the __karma_not_predicate__ combined with the
feature of many Karma primitive generators to /fail/ under certain conditions.
Here is the first alternative again for your reference:
!double_(0.0) << '(' << double_ << ", " << double_ << ')'
The generator `!double_(0.0)` does several things. First, because of the
__karma_not_predicate__, it succeeds only if the `double_(0.0)` generator
/fails/, making the whole first alternative fail otherwise. Second, the
`double_(0.0)` generator succeeds only if the value of its attribute is equal
to its immediate parameter (i.e. in this case `0.0`). And third, the
not predicate does not emit any output (regardless whether it succeeds or
fails), discarding any possibly emitted output from the `double_(0.0)`.
As we pass the imaginary part of the complex number as the attribute value for
the `!double_(0.0)`, the overall first alternative will be chosen only if
it is not equal to zero (the `!double_(0.0)` does not fail). That is exactly
what we need!
Now, the second alternative has to emit the real part of the complex
number only. In order to simplify the overall grammar we strive to unify the
attribute types of all alternatives. As the attribute type exposed by the first
alternative is `tuple<double, double, double>`, we need to skip the first and
last element of the attribute (remember, we pass the real part as the second
attribute element). We achieve this by using the `omit[]` directive:
omit[double_] << double_ << omit[double_]
The overall attribute of this expression is `tuple<double, double, double>`,
but the `omit[]` 'eats up' the first and the last element. The output emitted
by this expression consist of a single generated double representing the second
element of the tuple, i.e. the real part of our complex number.
[important Generally, it is preferable to use generator constructs not
requiring semantic actions. The reason is that semantic actions
often use constructs like: `double_[_1 = c.real()]`. But this
assignment is a real one! The data is in fact /copied/ to the
attribute value of the generator attached to the action. On the
other hand, grammars without any semantic actions usually don't
have to copy the attributes, making them more efficient.]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:karma_adapted_complex Complex - Fully Integrated]
[import ../../example/karma/complex_number_adapt.cpp]
Until now, we have been working around the fact that `std::complex<>` is not
a native __fusion__ sequence. We have not been able to use it with the same
simplicity and natural grace of a `fusion::tuple<>` or a similar __fusion__
data structure. Fortunately, starting with Boost V1.43 it is possible to
adapt any data structure (not only, as before, structures with publicly
accessible members) as a __fusion__ sequence. All we have to do is to employ one
of the new `BOOST_FUSION_ADAPT_ADT` macros.
[heading Adapting a Class As a Fusion Sequence]
Let us start with the code again, following up with the explanations afterwards.
Wouldn't it be optimal if we could pass our instance of a `std::complex<>`
directly to /Karma's/ `generate()` function:
[tutorial_karma_complex_number_adapt]
Indeed, this is possible! All we have to supply to make this work is a magic
incantation (somewhere in the global namespace):
[tutorial_karma_complex_number_adapt_class]
Most of the formatting grammar itself has not changed from the last section. We
still utilize a very similar scheme. We have an alternative providing the
formatting rules for our both use cases: one for the full complex format and
one for complex numbers with a zero imaginary part. But instead of selecting
the required alternative by comparing the imaginary part to zero in the grammar
we assume to receive a boolean attribute carrying this information:
&true_ << "(" << double_ << ", " << double_ << ")"
This reads as: 'if the first (boolean) element of the supplied fusion sequence
is `true`, proceed as specified, else select the next alternative'. The next
alternative now accounts for the boolean element as well, but is otherwise
(almost) unchanged from the last section's example.
Now it should be clear why our adapt construct above exposes a three element
__fusion__ sequence: a boolean and two double values (the real and the
imaginary part of the complex number). We want it to match the requirements of
our formatting grammar, which expects those exact values. The
`BOOST_FUSION_ADAPT_ADT` macro allows us to specify an arbitrary accessor
construct, not necessarily limited to just calling a member function of the
object instance (represented by `obj` in the context of this macro). This
allows us to nicely encapsulate the decision logic into the class adaptation.
Here is the last new bit of information. If you look closely you realize the
second alternative to be 'shorter' than the first one. It consumes only
two elements of the supplied fusion sequence: it ignores the boolean and uses
the real part of the complex number to generate its output. If there are more
elements in our attribute than needed, we now can safely omit them from the
grammar (which is a new 'feature' added to __spirit__ in V1.43 as well).
Note, we could have written the alternative as
&false_ << double_
but this would have been a bit less efficient as we needed to compare the
boolean value again, while the final solution provided will just ignore it.
[endsect]

View File

@@ -0,0 +1,435 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section Generator Concepts]
__karma__ generators fall into a couple of generalized __concepts__. The
/Generator/ is the most fundamental concept. All __karma__ generators are
models of the /Generator/ concept. /PrimitiveGenerator/, /UnaryGenerator/,
/BinaryGenerator/, /NaryGenerator/, and /Nonterminal/ are all refinements of
the /Generator/ concept.
The following sections provide details on these concepts.
[/////////////////////////////////////////////////////////////////////////////]
[section Generator]
[heading Description]
The /Generator/ is the most fundamental concept. A Generator has a member
function, `generate`, that accepts an `OutputIterator` and
returns bool as its result. The iterator receives the data being generated.
The Generator's `generate` member function returns `true` if the generator
succeeds. Each Generator can represent a specific pattern or algorithm, or it
can be a more complex generator formed as a composition of other Generators.
[variablelist Notation
[[`g`] [A `Generator`.]]
[[`G`] [A `Generator` type.]]
[[`OutIter`] [An `OutputIterator` type.]]
[[`sink`] [An `OutputIterator` instance.]]
[[`Context`] [The generator's __karma_context__ type.]]
[[`context`] [The generator's __karma_context__, or __unused__.]]
[[`delimit`] [A delimiter Generator, or __unused__.]]
[[`attrib`] [A __karma_compatible_attribute__, or __unused__.]]
]
[heading Valid Expressions]
In the expressions below, the behavior of the generator, `g`, as well as how
`delimit` and `attrib` are handled by `g`, are left unspecified in the base
`Generator` concept. These are specified in subsequent, more refined concepts
and by the actual models thereof.
For any Generator the following expressions must be valid:
[table
[[Expression] [Semantics] [Return type]]
[[
``g.generate(sink, context, delimit, attrib)``]
[Generate the output sequence by inserting the
generated characters/tokens into `sink`. Use the
`delimit` generator for delimiting. Return
`true` if successful, otherwise
return `false`.] [`bool`]]
[[`g.what(context)`] [Get information about a Generator.] [__info__]]
]
[heading Type Expressions]
[table
[[Expression] [Description]]
[[`G::template attribute<Context>::type`] [The Generator's attribute.]]
[[`traits::is_generator<G>::type`] [Metafunction that evaluates to `mpl::true_` if
a certain type, `G` is a Generator, `mpl::false_`
otherwise (See __mpl_boolean_constant__).]]
[[`G::properties`] [An `mpl::int_` (See __mpl_int_constant__) holding
a value from the `karma::generator_properties`
enumeration. The default value is
`generator_properties::no_properties`]]
]
[heading Postcondition]
Upon return from `g.generate` the following post conditions should hold:
* On successful generation, `sink` receives the generated characters/tokens
sequence.
* No pre-delimits: `delimit` characters/tokens will not be emitted in front of
any other output.
* The attribute `attrib` has not been modified.
[heading Models]
All generators in __karma__ are models of the /Generator/ concept.
[endsect] [/ Generator Concept]
[/////////////////////////////////////////////////////////////////////////////]
[section PrimitiveGenerator]
[heading Description]
/PrimitiveGenerator/ is the most basic building block that the client uses
to build more complex generators.
[heading Refinement of]
[:__generator_concept__]
[heading Post-delimit]
Before exiting the `generate` member function, a PrimitiveGenerator is required
to do a post-delimit. This will generate a single delimiting character/token
sequence. Only PrimitiveGenerator's are required to perform this post-delimit.
This is typically carried out through a call to `karma::delimit_out`:
karma::delimit_out(sink, delimit);
[heading Type Expressions]
[table
[[Expression] [Description]]
[[`traits::is_primitive_generator<G>::type`] [Metafunction that evaluates to `mpl::true_` if
a certain type, `G`, is a PrimitiveGenerator, `mpl::false_`
otherwise (See __mpl_boolean_constant__).]]
]
[heading Models]
The following generators conform to this model:
* __karma_eol__,
* __karma_eps__,
* [link spirit.karma.reference.numeric Numeric generators],
* [karma_char Character generators].
__fixme__ Add more links to /PrimitiveGenerator/ models here.
[endsect] [/ PrimitiveGenerator Concept]
[/////////////////////////////////////////////////////////////////////////////]
[section UnaryGenerator]
[heading Description]
/UnaryGenerator/ is a composite generator that has a single subject. The
UnaryGenerator may change the behavior of its subject following the
__delegate_pattern__.
[heading Refinement of]
[:__generator_concept__]
[variablelist Notation
[[`g`] [A UnaryGenerator.]]
[[`G`] [A UnaryGenerator type.]]
]
[heading Valid Expressions]
In addition to the requirements defined in __generator_concept__, for any
UnaryGenerator the following must be met:
[table
[[Expression] [Semantics] [Return type]]
[[`g.subject`] [Subject generator.] [__generator_concept__]]
]
[heading Type Expressions]
[table
[[Expression] [Description]]
[[`G::subject_type`] [The subject generator type.]]
[[`traits::is_unary_generator<G>::type`] [Metafunction that evaluates to `mpl::true_` if
a certain type, `G` is a UnaryGenerator, `mpl::false_`
otherwise (See __mpl_boolean_constant__).]]
]
[heading Invariants]
For any UnaryGenerator, `G`, the following invariant always holds:
* `traits::is_generator<G::subject_type>::type` evaluates to `mpl::true_`
[heading Models]
The following generators conform to this model:
* [karma_kleene Kleene Star (unary `*`)] operator,
* __karma_plus__ operator,
* __karma_optional__ operator,
* __karma_and_predicate__ and __karma_not_predicate__ operators,
* [karma_align `left_align`], [karma_align `center`], and [karma_align `right_align`] directives,
* [karma_repeat `repeat`] directive,
* __karma_verbatim__ directive,
* [karma_delimit `delimit`] directive,
* [karma_upperlower `lower`] and [karma_upperlower `upper`] directives,
* [karma_maxwidth `maxwidth`] directive,
* __karma_buffer__ directive,
* __karma_omit__ directive.
__fixme__ Add more links to models of UnaryGenerator concept
[endsect] [/ UnaryGenerator Concept]
[/////////////////////////////////////////////////////////////////////////////]
[section BinaryGenerator]
[heading Description]
/BinaryGenerator/ is a composite generator that has a two subjects, `left` and
`right`. The BinaryGenerator allows its subjects to be treated in the same
way as a single instance of a __generator_concept__ following the
__composite_pattern__.
[heading Refinement of]
[:__generator_concept__]
[variablelist Notation
[[`g`] [A BinaryGenerator.]]
[[`G`] [A BinaryGenerator type.]]
]
[heading Valid Expressions]
In addition to the requirements defined in __generator_concept__, for any
BinaryGenerator the following must be met:
[table
[[Expression] [Semantics] [Return type]]
[[`g.left`] [Left generator.] [__generator_concept__]]
[[`g.right`] [Right generator.] [__generator_concept__]]
]
[heading Type Expressions]
[table
[[Expression] [Description]]
[[`G::left_type`] [The left generator type.]]
[[`G::right_type`] [The right generator type.]]
[[`traits::is_binary_generator<G>::type`] [Metafunction that evaluates to `mpl::true_` if
a certain type, `G` is a BinaryGenerator, `mpl::false_`
otherwise (See __mpl_boolean_constant__).]]
]
[heading Invariants]
For any BinaryGenerator, `G`, the following invariants always hold:
* `traits::is_generator<G::left_type>::type` evaluates to `mpl::true_`
* `traits::is_generator<G::right_type>::type` evaluates to `mpl::true_`
[heading Models]
The following generators conform to this model:
* __karma_list__.
__fixme__ Add more links to models of BinaryGenerator concept
[endsect] [/ BinaryGenerator Concept]
[/////////////////////////////////////////////////////////////////////////////]
[section NaryGenerator]
[heading Description]
/NaryGenerator/ is a composite generator that has one or more subjects. The
NaryGenerator allows its subjects to be treated in the same way as a single
instance of a __generator_concept__ following the __composite_pattern__.
[heading Refinement of]
[:__generator_concept__]
[variablelist Notation
[[`g`] [A NaryGenerator.]]
[[`G`] [A NaryGenerator type.]]
]
[heading Valid Expressions]
In addition to the requirements defined in __generator_concept__, for any
NaryGenerator the following must be met:
[table
[[Expression] [Semantics] [Return type]]
[[`g.elements`] [The tuple of elements.] [A __fusion__ Sequence of __generator_concept__ types.]]
]
[heading Type Expressions]
[table
[[Expression] [Description]]
[[`g.elements_type`] [Elements tuple type.]]
[[`traits::is_nary_generator<G>::type`] [Metafunction that evaluates to `mpl::true_` if
a certain type, `G` is a NaryGenerator, `mpl::false_`
otherwise (See __mpl_boolean_constant__).]]
]
[heading Invariants]
For each element, `E`, in any NaryGenerator, `G`, the following
invariant always holds:
* `traits::is_generator<E>::type` evaluates to `mpl::true_`
[heading Models]
The following generators conform to this model:
* __karma_sequence__,
* __karma_alternative__.
__fixme__ Add more links to models of NaryGenerator concept
[endsect] [/ NaryGenerator Concept]
[/////////////////////////////////////////////////////////////////////////////]
[section Nonterminal]
[heading Description]
A Nonterminal is a symbol in a __peg__ production that represents a
grammar fragment. Nonterminals may self reference to specify recursion.
This is one of the most important concepts and the reason behind the
word "recursive" in recursive descent generation.
[heading Refinement of]
[:__generator_concept__]
[heading Signature]
Nonterminals can have both consumed and inherited attributes. The
Nonterminal's /Signature/ specifies both the consumed and inherited
attributes. The specification uses the function declarator syntax:
RT(A0, A1, A2, ..., AN)
where `RT` is the Nonterminal's consumed attribute and `A0` ... `AN`
are the Nonterminal's inherited attributes.
The default value is `void()` (no consumed and inherited attributes).
[heading Attributes]
The Nonterminal models a C++ function. The Nonterminal's consumed attribute is
analogous to the function return value as it is the type -exposed- by the
Nonterminal. Its inherited attributes are analogous to function arguments.
The inherited attributes (arguments) can be passed in just like any
__karma_lazy_argument__, e.g.:
r(expr) // Evaluate expr at parse time and pass the result to the Nonterminal r
[heading `_val`]
The `boost::spirit::karma::_val` placeholder can be used in __phoenix__
semantic actions anywhere in the Nonterminal's definition. This
__phoenix__ placeholder refers to the Nonterminal's (consumed)
attribute. The `_val` placeholder acts like an immutable reference to the
Nonterminal's attribute.
[note Starting with __spirit__ V2.5 (distributed with Boost V1.47) the
placeholder `_val` can be used in semantic actions attached to top level
generator components as well. See __generator_api__ for more information.]
[heading `_r1`...`r10`]
The `boost::spirit::_r1`...`boost::spirit::r10` placeholders can be used
in __phoenix__ semantic actions anywhere in the Nonterminal's
definition. These __phoenix__ placeholders refer to the Nonterminal's
inherited attributes.
[heading Locals]
Nonterminals can have local variables that will be created on the stack
at runtime. A locals descriptor added to the Nonterminal declaration
will give the Nonterminal local variables:
template <typename T0, typename T1, typename T2, ..., typename TN>
struct locals;
where `T0` ... `TN` are the types of local variables accessible in your
__phoenix__ semantic actions using the placeholders:
* `boost::spirit::_a`
* `boost::spirit::_b`
* `boost::spirit::_c`
* `boost::spirit::_d`
* `boost::spirit::_e`
* `boost::spirit::_f`
* `boost::spirit::_g`
* `boost::spirit::_h`
* `boost::spirit::_i`
* `boost::spirit::_j`
which correspond to the Nonterminal's local variables `T0` ... `T9`.
[variablelist Notation
[[`x`] [A Nonterminal]]
[[`X`] [A Nonterminal type]]
[[`arg1`, `arg2`, ..., `argN`] [__karma_lazy_arguments__ that evaluate to each of
the Nonterminal's inherited attributes.]]
]
[heading Valid Expressions]
In addition to the requirements defined in __generator_concept__, for any
Nonterminal the following must be met:
[table
[[Expression] [Semantics] [Return type]]
[[`x`] [In a generator expression, invoke Nonterminal `x`] [`X`]]
[[`x(arg1, arg2, ..., argN)`][In a generator expression, invoke Nonterminal `x`
passing in inherited attributes
`arg1`...`argN`] [`X`]]
[[`x.name(name)`] [Set the name of a Nonterminal] [`void`]]
[[`x.name()`] [Get the name of a Nonterminal] [`std::string`]]
]
[heading Type Expressions]
[table
[[Expression] [Description]]
[[`X::sig_type`] [The Signature of `X`: In a function signature form
as described above in the Signature paragraph.]]
[[`X::locals_type`] [The local variables of `X`: An __mpl_fwd_sequence__.]]
]
[heading Models]
* __karma_rule__
* __karma_grammar__
[endsect]
[endsect]

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,394 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:generate_api Generator API]
[//////////////////////////////////////////////////////////////////////////////]
[section:iterator_api Iterator Based Generator API]
[heading Description]
The library provides a couple of free functions to make generating a snap.
These generator functions have two forms. The first form, `generate`,
concatenates the output generated by the involved components without inserting
any output in between. The second `generate_delimited` intersperses the output
generated by the involved components using the given delimiter generator.
Both versions can take in attributes by (constant) reference that hold the
attribute values to output.
[heading Header]
// forwards to <boost/spirit/home/karma/generate.hpp>
#include <boost/spirit/include/karma_generate.hpp>
For variadic attributes:
// forwards to <boost/spirit/home/karma/generate_attr.hpp>
#include <boost/spirit/include/karma_generate_attr.hpp>
The variadic attributes version of the API allows one or more
attributes to be passed into the `generate` functions. The functions taking two
or more attributes are usable when the generator expression is a
__karma_sequence__ only. In this case each of the
attributes passed have to match the corresponding part of the sequence.
For the API functions deducing the correct (matching) generator type from the
supplied attribute type:
// forwards to <boost/spirit/home/karma/detail/generate_auto.hpp>
#include <boost/spirit/include/karma_generate_auto.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::karma::generate` ]]
[[`boost::spirit::karma::generate_delimited` ]]
[[`boost::spirit::karma::delimit_flag::predelimit` ]]
[[`boost::spirit::karma::delimit_flag::dont_predelimit` ]]
]
[heading Synopsis]
namespace boost { namespace spirit { namespace karma
{
template <typename OutputIterator, typename Expr>
inline bool
generate(
OutputIterator& sink
, Expr const& expr);
template <typename OutputIterator, typename Expr
, typename Attr1, typename Attr2, ..., typename AttrN>
inline bool
generate(
OutputIterator& sink
, Expr const& expr
, Attr1 const& attr1, Attr2 const& attr2, ..., AttrN const& attrN);
template <typename OutputIterator, typename Expr, typename Delimiter>
inline bool
generate_delimited(
OutputIterator& sink
, Expr const& expr
, Delimiter const& delimiter
, BOOST_SCOPED_ENUM(delimit_flag) pre_delimit = delimit_flag::dont_predelimit);
template <typename OutputIterator, typename Expr, typename Delimiter
, typename Attr1, typename Attr2, ..., typename AttrN>
inline bool
generate_delimited(
OutputIterator& sink
, Expr const& expr
, Delimiter const& delimiter
, Attr1 const& attr1, Attr2 const& attr2, ..., AttrN const& attrN);
template <typename OutputIterator, typename Expr, typename Delimiter
, typename Attr1, typename Attr2, ..., typename AttrN>
inline bool
generate_delimited(
OutputIterator& sink
, Expr const& expr
, Delimiter const& delimiter
, BOOST_SCOPED_ENUM(delimit_flag) pre_delimit
, Attr1 const& attr1, Attr2 const& attr2, ..., AttrN const& attrN);
}}}
[note Starting with __spirit__ V2.5 (distributed with Boost V1.47) the
placeholder `_val` can be used in semantic actions attached to top level
generator components. In this case `_val` refers to the supplied attribute
as a whole. For API functions taking more than one attribute argument
`_val` will refer to a Fusion vector or references to the attributes.]
__karma__ generator API functions based on the automatic creation of the
matching generator type:
namespace boost { namespace spirit { namespace karma
{
template <typename OutputIterator, typename Attr, typename Delimiter>
inline bool
generate_delimited(
OutputIterator& sink
, Attr const& attr
, Delimiter const& delimiter
, BOOST_SCOPED_ENUM(delimit_flag) pre_delimit = delimit_flag::dont_predelimit);
template <typename OutputIterator, typename Attr>
inline bool
generate(
OutputIterator& sink
, Attr const& attr);
}}}
All functions above return `true` if none of the involved generator components
failed, and `false` otherwise. If during the process of the output generation
the underlying output stream reports an error, the return value will be `false`
as well.
The maximum number of supported arguments is limited by the preprocessor
constant `SPIRIT_ARGUMENTS_LIMIT`. This constant defaults to the value defined
by the preprocessor constant `PHOENIX_LIMIT` (which in turn defaults to `10`).
[note The variadic functions with two or more attributes internally combine
(constant) references to all passed attributes into a `fusion::vector`
and forward this as a combined attribute to the corresponding function
taking one attribute.]
The `generate_delimited` functions not taking an explicit `delimit_flag` as one
of their arguments don't invoke the passed delimiter before starting to generate
output from the generator expression. This can be enabled by using the other
version of that function while passing `delimit_flag::predelimit` to the
corresponding argument.
[heading Template parameters]
[table
[[Parameter] [Description]]
[[`OutputIterator`] [__outputiter__ receiving the generated output.]]
[[`Expr`] [An expression that can be converted to a Karma generator.]]
[[`Delimiter`] [Generator used to delimit the output of the expression components.]]
[[`Attr`] [An attribute type utilized to create the corresponding
generator type from.]]
[[`Attr1`, `Attr2`, ..., `AttrN`][One or more attributes.]]
]
[endsect] [/ Iterator Based Generator API]
[//////////////////////////////////////////////////////////////////////////////]
[section:stream_api Stream Based Generator API]
[heading Description]
The library provides a couple of Standard IO __iomanip__ allowing to integrate
__karma__ output generation facilities with Standard output streams.
These generator manipulators have two forms. The first form, `format`,
concatenates the output generated by the involved components without inserting
any output in between. The second, `format_delimited`, intersperses the output
generated by the involved components using the given delimiter generator.
Both versions can take in attributes by (constant) reference that hold the
attribute values to output.
[heading Header]
// forwards to <boost/spirit/home/karma/stream/format_manip.hpp>
#include <boost/spirit/include/karma_format.hpp>
For variadic attributes:
// forwards to <boost/spirit/home/karma/stream/format_manip_attr.hpp>
#include <boost/spirit/include/karma_format_attr.hpp>
The variadic attributes version of the API allows one or more
attributes to be passed into the `format` manipulators. The manipulators taking
two or more attributes are usable when the generator expression is a
__karma_sequence__ only. In this case each of the attributes passed have to
match the corresponding part of the sequence.
For the API functions deducing the correct (matching) generator type from the
supplied attribute type:
// forwards to <boost/spirit/home/karma/format_auto.hpp>
#include <boost/spirit/include/karma_format_auto.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::karma::format` ]]
[[`boost::spirit::karma::format_delimited` ]]
[[`boost::spirit::karma::delimit_flag::predelimit` ]]
[[`boost::spirit::karma::delimit_flag::dont_predelimit` ]]
]
[heading Synopsis]
namespace boost { namespace spirit { namespace karma
{
template <typename Expr>
inline <unspecified>
format(
Expr const& xpr);
template <typename Expr
, typename Attr1, typename Attr2, ..., typename AttrN>
inline <unspecified>
format(
Expr const& xpr
, Attr1 const& attr1, Attr2 const& attr2, ..., AttrN const& attrN);
template <typename Expr, typename Delimiter>
inline <unspecified>
format_delimited(
Expr const& expr
, Delimiter const& d
, BOOST_SCOPED_ENUM(delimit_flag) pre_delimit = delimit_flag::dont_predelimit);
template <typename Expr, typename Delimiter
, typename Attr1, typename Attr2, ..., typename AttrN>
inline <unspecified>
format_delimited(
Expr const& expr
, Delimiter const& d
, Attr1 const& attr1, Attr2 const& attr2, ..., AttrN const& attrN);
template <typename Expr, typename Delimiter
, typename Attr1, typename Attr2, ..., typename AttrN>
inline <unspecified>
format_delimited(
Expr const& expr
, Delimiter const& d
, BOOST_SCOPED_ENUM(delimit_flag) pre_delimit
, Attr1 const& attr1, Attr2 const& attr2, ..., AttrN const& attrN);
}}}
__karma__ generator API functions based on the automatic creation of the
matching generator type:
namespace boost { namespace spirit { namespace karma
{
template <typename Attr, typename Delimiter>
inline <unspecified>
format_delimited(
Attr const& attr
, Delimiter const& d
, BOOST_SCOPED_ENUM(delimit_flag) pre_delimit = delimit_flag::dont_predelimit);
template <typename Attr>
inline <unspecified>
format(
Attr const& xpr);
}}}
All functions above return a standard IO stream manipulator instance (see
__iomanip__), which when streamed to an output stream will result in generating
the output as emitted by the embedded __karma__ generator expression. Any error
occurring during the invocation of the __karma__ generators will be reflected
in the streams status flag (`std::ios_base::failbit` will be set).
The maximum number of supported arguments is limited by the preprocessor
constant `SPIRIT_ARGUMENTS_LIMIT`. This constant defaults to the value defined
by the preprocessor constant `PHOENIX_LIMIT` (which in turn defaults to `10`).
[note The variadic manipulators with two or more attributes internally combine
(constant) references to all passed attributes into a `fusion::vector`
and forward this as a combined attribute to the corresponding manipulator
taking one attribute.]
The `format_delimited` manipulators not taking an explicit `delimit_flag` as one
of their arguments don't invoke the passed delimiter before starting to generate
output from the generator expression. This can be enabled by using the other
version of that manipulator while passing `delimit_flag::predelimit` to the
corresponding argument.
[heading Template parameters]
[table
[[Parameter] [Description]]
[[`Expr`] [An expression that can be converted to a Karma generator.]]
[[`Delimiter`] [Generator used to delimit the output of the expression components.]]
[[`Attr`] [An attribute type utilized to create the corresponding
generator type from.]]
[[`Attr1`, `Attr2`, ..., `AttrN`][One or more attributes.]]
]
[endsect] [/ Stream Based Generator API]
[//////////////////////////////////////////////////////////////////////////////]
[section:create_generator API for Automatic Generator Creation]
[heading Description]
The library implements a special API returning a generator instance for a
supplied attribute type. This function finds the best matching generator type
for the attribute based on a set of simple matching rules (as outlined in the
table below) applied recursively to the attribute type. The returned generator
can be utilized to emit output for the provided attribute.
[heading Header]
// forwards to <boost/spirit/home/karma/auto.hpp>
#include <boost/spirit/include/karma_auto.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::karma::create_generator`]]
[[`boost::spirit::traits::create_generator_exists`]]
]
[heading Synopsis]
namespace boost { namespace spirit { namespace karma
{
template <typename Attr>
inline <unspecified>
create_generator();
}}}
The returned instance can be directly passed as the generator (or the
delimiting generator) to any of the __karma__ API functions. Additionally it
can be assigned to a rule as the rules right hand side expression. This
function will return a valid generator type only if the meta function
`traits::create_generator_exists` returns `mpl::true_`. Otherwise it will fail
compiling.
namespace boost { namespace spirit { namespace traits
{
template <typename Attr>
struct create_generator_exists;
}}}
The meta function evaluates to `mpl::true_` if `create_generator` would return
a valid generator for the given type `Attr`.
The following table outlines the mapping rules from the attribute type to the
generator type. These rules are applied recursively to create the generator
type which can be used to generate output from the given attribute type.
[table
[[Attribute type] [Generator type]]
[[`char`, `wchar_t`] [`standard::char_`, `standard_wide::char_`]]
[[`short`, `int`, `long`] [`short_`, `int_`, `long_`]]
[[`unsigned short`, `unsigned int`, `unsigned long`]
[`ushort_`, `uint_`, `ulong_`]]
[[`float`, `double`, `long double`] [`float_`, `double_`, `long_double`]]
[[`short`, `int`, `long`] [`short_`, `int_`, `long_`]]
[[`long long`, `unsigned long long`]
[`long_long`, `ulong_long`]]
[[`bool`] [`bool_`]]
[[Any string (`char const*`, `std::string`, etc.)]
[`string`]]
[[Any (STL) container] [Kleene Star (unary `'*'`)]]
[[Any Fusion sequence] [Sequence operator (`'<<'`)]]
[[`boost::optional<>`] [Optional operator (unary `'-'`)]]
[[`boost::variant<>`] [Alternative operator (`'|'`)]]
]
[important The mapping for the generators `long_long` and `ulong_long` are only
available on platforms where the preprocessor constant
`BOOST_HAS_LONG_LONG` is defined (i.e. on platforms having native
support for `long long` and `unsigned long long` (64 bit) signed and
unsigned integer types).]
[heading Template parameters]
[table
[[Parameter] [Description]]
[[`Attr`] [An attribute type utilized to create the corresponding
generator type from.]]
]
[endsect] [/ API for Automatic Generator Creation]
[endsect]

View File

@@ -0,0 +1,267 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:nonterminal Nonterminal Generators]
[heading Module Headers]
// forwards to <boost/spirit/home/karma/nonterminal.hpp>
#include <boost/spirit/include/karma_nonterminal.hpp>
Also, see __include_structure__.
[//////////////////////////////////////////////////////////////////////////////]
[section:rule Generator Rule]
[heading Description]
The rule is a polymorphic generator that acts as a named place-holder
capturing the behavior of a PEG expression assigned to it. Naming a
__peg__ expression allows it to be referenced later and makes it
possible for the rule to call itself. This is one of the most important
mechanisms and the reason behind the word "recursive" in recursive
descent output generation.
[heading Header]
// forwards to <boost/spirit/home/karma/nonterminal/rule.hpp>
#include <boost/spirit/include/karma_rule.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::karma::rule`]]
]
[heading Synopsis]
template <typename OutputIterator, typename A1, typename A2, typename A3>
struct rule;
[heading Template parameters]
[table
[[Parameter] [Description] [Default]]
[[`OutputIterator`] [The underlying output iterator
type that the rule is
expected to work on.] [none]]
[[`A1`, `A2`, `A3`] [Either `Signature`,
`Delimiter` or `Locals` in
any order. See table below.] [See table below.]]
]
Here is more information about the template parameters:
[table
[[Parameter] [Description] [Default]]
[[`Signature`] [Specifies the rule's consumed
(value to output) and inherited
(arguments) attributes. More on
this here: __karma_nonterminal_concept__.]
[__unused_type__.
When `Signature` defaults
to __unused_type__, the effect
is the same as specifying a signature
of `void()` which is also equivalent
to `unused_type()`]]
[[`Delimiter`] [Specifies the rule's delimiter
generator. Specify this if you
want the rule to delimit the
generated output.] [__unused_type__]]
[[`Locals`] [Specifies the rule's local
variables. See __karma_nonterminal_concept__.]
[__unused_type__]]
]
[heading Model of]
[:__karma_nonterminal_concept__]
[variablelist Notation
[[`r, r2`] [Rules]]
[[`g`] [A generator expression]]
[[`OutputIterator`] [The underlying output iterator type that the rule is
expected to work on.]]
[[`A1`, `A2`, `A3`] [Either `Signature`, `Delimiter` or `Locals` in
any order.]]
]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __karma_nonterminal_concept__.
[table
[[Expression] [Description]]
[[
``rule<OutputIterator, A1, A2, A3>
r(name);``]
[Rule declaration. `OutputIterator` is required.
`A1, A2, A3` are optional and can be specified in any order.
`name` is an optional string that gives the rule
its name, useful for debugging.]]
[[
``rule<OutputIterator, A1, A2, A3>
r(r2);``] [Copy construct rule `r` from rule `r2`.]]
[[`r = r2;`] [Assign rule `r2` to `r`.]]
[[`r.alias()`] [Return an alias of `r`. The alias is a generator that
holds a reference to `r`. Reference semantics.]]
[[`r.copy()`] [Get a copy of `r`.]]
[[`r = g;`] [Rule definition]]
[[`r %= g;`] [Auto-rule definition. The attribute of `g` should be
compatible with the consumed attribute of `r`.]]
[[`r.name()`] [Retrieve the current name of the rule object.]]
[[`r.name(name)`] [Set the current name of the rule object to be `name`.]]
]
[heading Attributes]
[:The rule's generator attribute is `RT`: The consumed attribute of the
rule. See __karma_nonterminal_attribute__]
[heading Complexity]
[:The complexity is defined by the complexity of the RHS generator, `g`]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
[karma_reference_rule]
[endsect] [/ Rule]
[////////////////////////////////////////////////////////////////////////////////]
[section:grammar Generator Grammar]
[heading Description]
The grammar encapsulates a set of __karma_rules__ (as well as primitive
generators (__primitive_generator_concept__) and sub-grammars). The grammar is
the main mechanism for modularization and composition. Grammars can be
composed to form more complex grammars.
[heading Header]
// forwards to <boost/spirit/home/karma/nonterminal/grammar.hpp>
#include <boost/spirit/include/karma_grammar.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::karma::grammar`]]
]
[heading Synopsis]
template <typename OutputIterator, typename A1, typename A2, typename A3>
struct grammar;
[heading Template parameters]
[table
[[Parameter] [Description] [Default]]
[[`OutputIterator`] [The underlying output iterator
type that the rule is
expected to work on.] [none]]
[[`A1`, `A2`, `A3`] [Either `Signature`,
`Delimiter` or `Locals` in
any order. See table below.] [See table below.]]
]
Here is more information about the template parameters:
[table
[[Parameter] [Description] [Default]]
[[`Signature`] [Specifies the grammar's synthesized
(return value) and inherited
attributes (arguments). More on
this here: __karma_nonterminal_concept__.]
[__unused_type__.
When `Signature` defaults
to __unused_type__, the effect
is the same as specifying a signature
of `void()` which is also equivalent
to `unused_type()`]]
[[`Delimiter`] [Specifies the grammar's delimiter
generator. Specify this if you
want the grammar to delimit the
generated output.] [__unused_type__]]
[[`Locals`] [Specifies the grammar's local
variables. See __karma_nonterminal_concept__.]
[__unused_type__]]
]
[heading Model of]
[:__karma_nonterminal_concept__]
[variablelist Notation
[[`g`] [A grammar]]
]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is not
defined in __karma_nonterminal_concept__.
[table
[[Expression] [Semantics]]
[[
``
template <typename OutputIterator>
struct my_grammar : grammar<OutputIterator, A1, A2, A3>
{
my_grammar() : my_grammar::base_type(start, name)
{
// Rule definitions
start = /* ... */;
}
rule<OutputIterator, A1, A2, A3> start;
// more rule declarations...
};
``
] [Grammar definition. `name` is an optional string that gives the
grammar its name, useful for debugging.]]
]
[note The template parameters of a grammar and its start rule (the rule passed
to the grammar's base class constructor) must match, otherwise you will
see compilation errors.]
[heading Attributes]
[:The generator attribute of the grammar is `RT`, its consumed attribute. See
__karma_nonterminal_attribute__]
[heading Complexity]
[:The complexity is defined by the complexity of the its definition.]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
[karma_reference_grammar_using]
[karma_reference_grammar_definition]
[karma_reference_grammar]
[endsect] [/ Grammar]
[endsect]

View File

@@ -0,0 +1,108 @@
[/==============================================================================
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2001-2011 Joel de Guzman
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[/////////////////////////////////////////////////////////////////////////////]
[section:num_list Number List - Printing Numbers From a std::vector]
[heading Using the List Operator]
The C++ Standard library lacks an important feature, namely the support for
any formatted output of containers. Sure, it's fairly easy to write a custom
routine to output a specific container, but doing so over and over again is
tedious at best. In this section we will demonstrate some more of the
capabilities of __karma__ for generating output from arbitrary STL containers.
We will build on the example presented in an earlier section (see
[link spirit.karma.tutorials.warming_up Warming Up]).
The full source code of the example shown in this section can be found here:
[@../../example/karma/num_list2.cpp num_list2.cpp].
[import ../../example/karma/num_list2.cpp]
This time we take advantage of Karma's __karma_list__ operator. The semantics
of the list operator are fully equivalent to the semantics of the sequence
we used before. The generator expression
double_ << *(',' << double_)
is semantically equivalent to the generator expression
double_ % ','
simplifying the overall code. The list operator's attribute is compatible with
any STL container as well. For a change we use a `std::vector<double>`
instead of the `std::list<double>` we used before. Additionally, the routine
`generate_numbers` takes the container as a template parameter, so it will now
work with any STL container holding `double` numbers.
[tutorial_karma_numlist2]
[note Despite the container being a template parameter, the __karma__
formatting expression (`double_ % ','`) does not depend on the actual
type of the passed container. The only precondition to be met here is
that the elements stored in the container have to be convertible to
`double`.]
[heading Generate Output from Arbitrary Data]
The output routine developed above is still not generically usable for all types
of STL containers and for arbitrary elements stored in them. In order to be
usable the items stored in the container still need to be convertible to a
`double`. Fortunately __karma__ is capable to output arbitrary
data types while using the same format description expression. It implements
the [karma_stream `stream`] generators which are able to consume any attribute
type as long as a matching standard streaming operator is defined. I.e.
for any attribute type `Attrib` a function:
std::ostream& operator<< (std::ostream&, Attrib const&);
needs to be available. The [karma_stream `stream`] generator will use the
standard streaming operator to generate the output.
The following example modifies the code shown above to utilize the
[karma_stream `stream`] operator, which makes it compatible with almost any
data type. We implement a custom data type `complex` to demonstrate this. The
example shows how it is possible to integrate this (or any other) custom data
type into the __karma__ generator framework.
[import ../../example/karma/num_list3.cpp]
This is the custom data structure together with the required standard streaming
operator:
[tutorial_karma_numlist3_complex]
And this is the actual call to generate the output from a vector of those. This
time we interleave the generated output with newline breaks (see
__karma_eol__), putting each complex number onto a separate line:
[tutorial_karma_numlist3]
The code shown is fully generic and can be used with any STL container as long
as the data items stored in that container implement the standard streaming
operator.
The full source code of the example presented in this section can be found here:
[@../../example/karma/num_list3.cpp num_list3.cpp].
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:num_matrix Matrix of Numbers - Printing Numbers From a Matrix]
In this section we will discuss the possibilities of __karma__ when it comes to
generating output from more complex - but still regular - data structures.
For simplicity we will use a `std::vector<std::vector<int> >` as a poor
man's matrix representation. But even if the data structure seems to be very
simple, the presented principles are applicable to more complex, or custom
data structures as well. The full source code of the example discussed in this
section can be found here: [@../../example/karma/num_matrix.cpp num_matrix.cpp].
[import ../../example/karma/num_matrix.cpp]
[endsect]

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,169 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:numeric_performance Performance of Numeric Generators]
[section:int_performance Comparing the performance of a single int_ generator]
These performance measurements are centered around default formatting of a
single `int` integer number using different libraries and methods.
The overall execution times for those examples are compared below. We compare
using `sprintf`, C++ iostreams, __boost_format__, and __karma__.
For the full source code of the performance test please see here:
[@../../workbench/karma/int_generator.cpp int_generator.cpp]. All the
measurements have been done by executing `1e7` iterations for each
formatting type (NUMITERATIONS is set to `1e7` in the code shown below).
[import ../../workbench/karma/int_generator.cpp]
Code used to measure the performance for `ltoa`:
[karma_int_performance_ltoa]
Code used to measure the performance for standard C++ iostreams:
[karma_int_performance_iostreams]
Code used to measure the performance for __boost_format__:
[karma_int_performance_format]
Code used to measure the performance for __karma__ using a plain character buffer:
[karma_int_performance_plain]
The following table shows the overall performance results collected
while using different compilers. All times are in seconds measured for `1e7`
iterations (platform: Windows7, Intel Core Duo(tm) Processor, 2.8GHz, 4GByte RAM).
For a more readable comparison of the results see this
[link spirit.karma.int_performance figure].
[table Performance comparison for a single int (all times in [s], `1e7` iterations)
[[Library] [gcc 4.4.0 (32 bit)] [VC++ 10 (32 bit)] [Intel 11.1 (32 bit)] [gcc 4.4.0 (64 bit)] [VC++ 10 (64 bit)] [Intel 11.1 (64 bit)]]
[[ltoa] [1.542] [0.895] [0.884] [1.163] [1.099] [0.906]]
[[iostreams] [6.548] [13.727] [11.898] [3.464] [8.316] [8.115]]
[[__boost_format__] [16.998] [21.813] [20.477] [17.464] [14.662] [13.646]]
[[__karma__ int_] [1.421] [0.744] [0.697] [1.072] [0.953] [0.606]]
]
[fig int_performance.png..Performance comparison for a single int..spirit.karma.int_performance]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:double_performance Comparing the performance of a single double_ generator]
These performance measurements are centered around default formatting of a
single `double` floating point number using different libraries and methods.
The overall execution times for those examples are compared below. We compare
using `sprintf`, C++ iostreams, __boost_format__, and __karma__.
For the full source code of the performance test please see here:
[@../../workbench/karma/double_performance.cpp double_performance.cpp]. All the
measurements have been done by executing `1e6` iterations for each
formatting type (NUMITERATIONS is set to `1e6` in the code shown below).
[import ../../workbench/karma/double_performance.cpp]
Code used to measure the performance for `sprintf`:
[karma_double_performance_printf]
Code used to measure the performance for standard C++ iostreams:
[karma_double_performance_iostreams]
Code used to measure the performance for __boost_format__:
[karma_double_performance_format]
The following code shows the common definitions used by all __karma__ performance
measurements as listed below:
[karma_double_performance_definitions]
Code used to measure the performance for __karma__ using a plain character buffer:
[karma_double_performance_plain]
The following table shows the overall performance results collected
while using different compilers. All times are in seconds measured for `1e6`
iterations (platform: Windows7, Intel Core Duo(tm) Processor, 2.8GHz, 4GByte RAM).
For a more readable comparison of the results see this
[link spirit.karma.double_performance figure].
[table Performance comparison for a single double (all times in [s], `1e6` iterations)
[[Library] [gcc 4.4.0 (32 bit)] [VC++ 10 (32 bit)] [Intel 11.1 (32 bit)] [gcc 4.4.0 (64 bit)] [VC++ 10 (64 bit)] [Intel 11.1 (64 bit)]]
[[sprintf] [0.755] [0.965] [0.880] [0.713] [0.807] [0.694]]
[[iostreams] [2.316] [2.624] [1.964] [1.634] [1.468] [1.354]]
[[__boost_format__] [3.188] [3.737] [2.878] [3.217] [2.672] [2.011]]
[[__karma__ double_] [0.813] [0.561] [0.368] [0.426] [0.260] [0.218]]
]
[fig double_performance.png..Performance comparison for a single double..spirit.karma.double_performance]
[endsect]
[////////////////////////////////////////////////////////////////////////////]
[section:format_performance Comparing the performance of a sequence of several generators]
These performance measurements are centered around formatting of a sequence of
different items, including 2 `double` floating point numbers using different
libraries and methods. The overall execution times for those examples are
compared below. We compare using `sprintf`, C++ iostreams, __boost_format__,
and __karma__.
For the full source code of the performance test please see here:
[@../../workbench/karma/format_performance.cpp format_performance.cpp]. All the
measurements have been done by doing `1e6` iterations for each formatting
type (NUMITERATIONS is set to `1e6`).
[import ../../workbench/karma/format_performance.cpp]
Code used to measure the performance for sprintf:
[karma_format_performance_printf]
Code used to measure the performance for standard iostreams:
[karma_format_performance_iostreams]
Code used to measure the performance for __boost_format__:
[karma_format_performance_format]
The following code shows the common definitions used by all __karma__
performance measurements as listed below:
[karma_format_performance_definitions]
Code used to measure the performance for __karma__ using a plain character
buffer:
[karma_format_performance_plain]
The following table shows the overall performance results collected
while using different compilers. All times are in seconds measured for `1e6`
iterations (platform: Windows7, Intel Core Duo(tm) Processor, 2.8GHz, 4GByte RAM).
For a more readable comparison of the results see this
[link spirit.karma.format_performance figure].
[table Performance comparison for a sequence of several items (all times in [s], `1e6` iterations)
[[Library] [gcc 4.4.0 (32 bit)] [VC++ 10 (32 bit)] [Intel 11.1 (32 bit)] [gcc 4.4.0 (64 bit)] [VC++ 10 (64 bit)] [Intel 11.1 (64 bit)]]
[[sprintf] [1.725] [1.892] [1.903] [1.469] [1.608] [1.493]]
[[iostreams] [4.827] [5.287] [4.444] [3.112] [3.319] [2.877]]
[[__boost_format__] [5.881] [7.089] [5.801] [5.455] [5.254] [4.164]]
[[__karma__] [1.942] [1.242] [0.999] [1.334] [0.758] [0.686]]
]
[fig format_performance.png..Performance comparison for a sequence of several items..spirit.karma.format_performance]
[endsect]
[endsect]

View File

@@ -0,0 +1,816 @@
[/==============================================================================
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2001-2011 Joel de Guzman
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:operator Generator Operators]
Operators are used as a means for object composition and embedding.
Simple generators may be composed to form composites through operator
overloading, crafted to approximate the syntax of __peg__ (PEG). An
expression such as:
a | b
yields a new generator type which is a composite of its operands, `a` and
`b`.
This module includes different generators which get instantiated if one of the
overloaded operators is used with more primitive generator constructs. It
includes sequences (`a << b`), alternatives (`a | b`), Kleene star (unary `*`),
plus (unary `+`), optional (unary `-`), lists (`a % b`), and the two predicates, the
/and/ predicate (unary `&`) and the /not/ predicate (unary `!`).
[heading Module Header]
// forwards to <boost/spirit/home/karma/operator.hpp>
#include <boost/spirit/include/karma_operator.hpp>
Also, see __include_structure__.
[/////////////////////////////////////////////////////////////////////////////]
[section:sequence Sequence Generator (`a << b`)]
[heading Description]
Generator sequences are used to consecutively combine different, more primitive
generators. All generators in a sequence are invoked from left to right as long
as they succeed.
[heading Header]
// forwards to <boost/spirit/home/karma/operator/sequence.hpp>
#include <boost/spirit/include/karma_sequence.hpp>
Also, see __include_structure__.
[heading Model of]
[:__nary_generator_concept__]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is not
defined in __nary_generator_concept__.
[table
[[Expression] [Semantics]]
[[`a << b`] [The generators `a` and `b` are executed sequentially
from left to right and as long as they succeed. A
failed generator stops the execution of the entire
sequence and makes the sequence fail as well.]]
]
It is important to note, that sequences don't perform any buffering of the
output generated by its elements. That means that any failing sequence might
have already generated some output, which is /not/ rolled back.
[tip The simplest way to force a sequence to behave as if it did buffering
is to wrap it into a buffering directive (see __karma_buffer__):
``buffer[a << b << c]``
which will /not/ generate any output in case of a failing sequence.]
[heading Attributes]
See __karma_comp_attr_notation__.
[table
[[Expression] [Attribute]]
[[`a << b` (sequence)]
[``a: A, b: B --> (a << b): tuple<A, B>
a: A, b: Unused --> (a << b): A
a: Unused, b: B --> (a << b): B
a: Unused, b: Unused --> (a << b): Unused
a: A, b: A --> (a << b): vector<A>
a: vector<A>, b: A --> (a << b): vector<A>
a: A, b: vector<A> --> (a << b): vector<A>
a: vector<A>, b: vector<A> --> (a << b): vector<A>``]]
]
[important The table above uses `tuple<A, B>` and `vector<A>` as placeholders
only.
The notation `tuple<A, B>` stands for /any fusion sequence of two
elements/, where `A` is the type of its first element and `B` is the
type of its second element.
The notation of `vector<A>` stands for /any STL container/ holding
elements of type `A`.]
The attribute composition and propagation rules as shown in the table above make
sequences somewhat special as they can operate in two modes if all elements have
the same attribute type: consuming fusion sequences and consuming STL
containers. The selected mode depends on the type of the attribute supplied.
[heading Complexity]
[:The overall complexity of the sequence generator is defined by the sum of the
complexities of its elements. The complexity of the sequence itself is O(N),
where N is the number of elements in the sequence.]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_sequence]
Basic usage of a sequence:
[reference_karma_sequence]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:alternative Alternative Generator (`a | b`)]
[heading Description]
Generator alternatives are used to combine different, more primitive generators
into alternatives. All generators in an alternative are invoked from left to
right until one of them succeeds.
[heading Header]
// forwards to <boost/spirit/home/karma/operator/alternative.hpp>
#include <boost/spirit/include/karma_alternative.hpp>
Also, see __include_structure__.
[heading Model of]
[:__nary_generator_concept__]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is not
defined in __nary_generator_concept__.
[table
[[Expression] [Semantics]]
[[`a | b`] [The generators `a` and `b` are executed sequentially
from left to right until one of them succeeds. A
failed generator forces the alternative generator to
try the next one. The alternative fails as a whole
only if all elements of the alternative fail. Each
element of the alternative gets passed the whole
attribute of the alternative.]]
]
Alternatives intercept and buffer the output of the currently executed element.
This allows to avoid partial outputs from failing elements as the buffered
content will be forwarded to the actual output only after an element succeeded.
[heading Attributes]
See __karma_comp_attr_notation__.
[table
[[Expression] [Attribute]]
[[`a | b` (alternative)]
[``a: A, b: B --> (a | b): variant<A, B>
a: A, b: Unused --> (a | b): A
a: Unused, b: B --> (a | b): B
a: Unused, b: Unused --> (a | b): Unused
a: A, b: A --> (a | b): A``]]
]
[important The table above uses `variant<A, B>` as a placeholder only. The
notation `variant<A, B>` stands for the type `boost::variant<A, B>`.
]
The attribute handling of Alternatives is special as their behavior is
not completely defined at compile time. First of all the selected alternative
element depends on the actual type of the attribute supplied to the alternative
generator (i.e. what is stored in the variant). The attribute type supplied at
/runtime/ narrows the set of considered alternatives to those being compatible
attribute wise. The remaining alternatives are tried sequentially until the
first of them succeeds. See below for an example of this behavior.
[heading Complexity]
[:The overall complexity of the alternative generator is defined by the sum of
the complexities of its elements. The complexity of the alternative itself is
O(N), where N is the number of elements in the alternative.]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_alternative]
Basic usage of an alternative. While being only the second alternative, the
`double_` generator is chosen for output formatting because the supplied
attribute type is not compatible (i.e. not convertible) to the attribute type
of the `string` alternative.
[reference_karma_alternative1]
The same formatting rules may be used to output a string. This time we supply
the string `"example"`, resulting in the first alternative to be chosen for the
generated output.
[reference_karma_alternative2]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:kleene Kleene Star Generator (`*a`)]
[heading Description]
Kleene star generators are used to repeat the execution of an embedded generator
zero or more times. Regardless of the success of the embedded generator, the
Kleene star generator always succeeds.
[heading Header]
// forwards to <boost/spirit/home/karma/operator/kleene.hpp>
#include <boost/spirit/include/karma_kleene.hpp>
Also, see __include_structure__.
[heading Model of]
[:__unary_generator_concept__]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is not
defined in __unary_generator_concept__.
[table
[[Expression] [Semantics]]
[[`*a`] [The generator `a` is executed zero or more times
depending on the availability of an attribute. The
execution of `a` stops after the attribute values
passed to the Kleene star generator are exhausted.
The Kleene star always succeeds (unless the
underlying output stream reports an error).]]
]
[note All failing iterations of the embedded generator will consume one element
from the supplied attribute.]
[heading Attributes]
See __karma_comp_attr_notation__.
[table
[[Expression] [Attribute]]
[[`*a` (Kleene star, unary `*`)]
[``a: A --> *a: vector<A>
a: Unused --> *a: Unused``]]
]
[important The table above uses `vector<A>` as a placeholder only. The notation
of `vector<A>` stands for /any STL container/ holding elements of
type `A`.]
The Kleene star generator will execute its embedded generator once for each
element in the provided container attribute as long as the embedded
generator succeeds. On each iteration it will pass the next consecutive element
from the container attribute to the embedded generator. Therefore the number of
iterations will not be larger than the number of elements in the container
passed as its attribute. An empty container will make the Kleene star
generate no output at all.
It is important to note, that the Kleene star does not perform any buffering
of the output generated by its embedded elements. That means that any failing
element generator might have already generated some output, which is /not/
rolled back.
[tip The simplest way to force a Kleene star to behave as if it did
buffering is to wrap it into a buffering directive (see
__karma_buffer__):
``buffer[*a]``
which will /not/ generate any output in case of a failing generator `*a`.
The expression:
``*(buffer[a])``
will not generate any partial output from a generator `a` if it fails
generating in the middle of its output. The overall expression will
still generate the output as produced by all successful invocations of
the generator `a`.]
[heading Complexity]
[:The overall complexity of the Kleene star generator is defined by the
complexity of its embedded generator multiplied by the number of executed
iterations. The complexity of the Kleene star itself is O(N), where N is the
number of elements in the container passed as its attribute.]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_kleene]
Basic usage of a Kleene star generator:
[reference_karma_kleene]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:plus Plus Generator (`+a`)]
[heading Description]
The Plus generator is used to repeat the execution of an embedded generator
one or more times. It succeeds if the embedded generator has been successfully
executed at least once.
[heading Header]
// forwards to <boost/spirit/home/karma/operator/plus.hpp>
#include <boost/spirit/include/karma_plus.hpp>
Also, see __include_structure__.
[heading Model of]
[:__unary_generator_concept__]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is not
defined in __unary_generator_concept__.
[table
[[Expression] [Semantics]]
[[`+a`] [The generator `a` is executed one or more times
depending on the availability of an attribute. The
execution of `a` stops after the attribute values
passed to the plus generator are exhausted.
The plus generator succeeds as long as its embedded
generator has been successfully executed at least once
(unless the underlying output stream reports an
error).]]
]
[note All failing iterations of the embedded generator will consume one element
from the supplied attribute. The overall `+a` will succeed as long as at
least one invocation of the embedded generator will succeed (unless the
underlying output stream reports an error).]
[heading Attributes]
See __karma_comp_attr_notation__.
[table
[[Expression] [Attribute]]
[[`+a` (unary `+`)]
[``a: A --> +a: vector<A>
a: Unused --> +a: Unused``]]
]
[important The table above uses `vector<A>` as a placeholder only. The notation
of `vector<A>` stands for /any STL container/ holding elements of
type `A`.]
The Plus generator will execute its embedded generator once for each
element in the provided container attribute as long as the embedded
generator succeeds. On each iteration it will pass the next consecutive element
from the container attribute to the embedded generator. Therefore the number of
iterations will not be larger than the number of elements in the container
passed as its attribute. An empty container will make the plus generator fail.
It is important to note, that the plus generator does not perform any buffering
of the output generated by its embedded elements. That means that any failing
element generator might have already generated some output, which is /not/
rolled back.
[tip The simplest way to force a plus generator to behave as if it did
buffering is to wrap it into a buffering directive (see
__karma_buffer__):
``buffer[+a]``
which will /not/ generate any output in case of a failing generator `+a`.
The expression:
``+(buffer[a])``
will not generate any partial output from a generator `a` if it fails
generating in the middle of its output. The overall expression will
still generate the output as produced by all successful invocations of
the generator `a`.]
[heading Complexity]
[:The overall complexity of the plus generator is defined by the
complexity of its embedded generator multiplied by the number of executed
iterations. The complexity of the plus generator itself is O(N), where N is
the number of elements in the container passed as its attribute.]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_plus]
Basic usage of a plus generator:
[reference_karma_plus1]
A more sophisticated use case showing how to leverage the fact that plus is
failing for empty containers passed as its attribute:
[reference_karma_plus2]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:list List Generator (`a % b`)]
[heading Description]
The list generator is used to repeat the execution of an embedded generator
and intersperse it with the output of another generator one or more times.
It succeeds if the embedded generator has been successfully executed at least
once.
[heading Header]
// forwards to <boost/spirit/home/karma/operator/list.hpp>
#include <boost/spirit/include/karma_list.hpp>
Also, see __include_structure__.
[heading Model of]
[:__binary_generator_concept__]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is not
defined in __binary_generator_concept__.
[table
[[Expression] [Semantics]]
[[`a % b`] [The generator `a` is executed one or more times
depending on the availability of an attribute. The
output generated by `a` is interspersed with the output
generated by `b`. The list generator succeeds if
its first embedded generator has been
successfully executed at least once (unless the
underlying output stream reports an error).]]
]
The list expression `a % b` is a shortcut for `a << *(b << a)`. It is almost
semantically equivalent, except for the attribute of `b`, which gets ignored
in the case of the list generator.
[note All failing iterations of the embedded generator will consume one element
from the supplied attribute. The overall `a % b` will succeed as long as at
least one invocation of the embedded generator, `a`, will succeed (unless
the underlying output stream reports an error).]
[heading Attributes]
See __karma_comp_attr_notation__.
[table
[[Expression] [Attribute]]
[[`a % b` (list)]
[``a: A, b: B --> (a % b): vector<A>
a: Unused, b: B --> (a % b): Unused``]]
]
[important The table above uses `vector<A>` as a placeholder only. The notation
of `vector<A>` stands for /any STL container/ holding elements of
type `A`.]
The list generator will execute its embedded generator once for each
element in the provided container attribute and as long as the embedded
generator succeeds. The output generated by its first generator will be
interspersed by the output generated by the second generator. On each iteration
it will pass the next consecutive element from the container attribute to the
first embedded generator. The second embedded generator does not get passed
any attributes (it gets invoked using an `unused_type` as its attribute).
Therefore the number of iterations will not be larger than the number of
elements in the container passed as its attribute. An empty container will make
the list generator fail.
[tip If you want to use the list generator and still allow for an empty
attribute, you can use the optional operator (see __karma_optional__):
``-(a % b)``
which will succeed even if the provided container attribute does not
contain any elements.
]
[heading Complexity]
[:The overall complexity of the list generator is defined by the
complexity of its embedded generators multiplied by the number of executed
iterations. The complexity of the list generator itself is O(N), where N is
the number of elements in the container passed as its attribute.]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_list]
Basic usage of a list generator:
[reference_karma_list]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:optional Optional Generator (`-a`)]
[heading Description]
The optional generator is used to conditionally execute an embedded generator.
It succeeds always.
[heading Header]
// forwards to <boost/spirit/home/karma/operator/optional.hpp>
#include <boost/spirit/include/karma_optional.hpp>
Also, see __include_structure__.
[heading Model of]
[:__unary_generator_concept__]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is not
defined in __unary_generator_concept__.
[table
[[Expression] [Semantics]]
[[`-a`] [The generator `a` is executed depending on the
availability of an attribute. The optional generator
succeeds if its embedded generator succeeds
(unless the underlying output stream reports an
error).]]
]
[heading Attributes]
See __karma_comp_attr_notation__.
[table
[[Expression] [Attribute]]
[[`-a` (optional, unary `-`)]
[``a: A --> -a: optional<A>
a: Unused --> -a: Unused``]]
]
[important The table above uses `optional<A>` as a placeholder only. The
notation of `optional<A>` stands for the data type
`boost::optional<A>`.]
The optional generator will execute its embedded generator once if the provided
attribute holds a valid value. It forwards the value held in its attribute
to the embedded generator.
It is important to note, that the optional generator does not perform any
buffering of the output generated by its embedded elements. That means that any
failing element might have already generated some output, which is /not/
rolled back.
[tip The simplest way to force a optional generator to behave as if it did
buffering is to wrap it into a buffering directive (see
__karma_buffer__):
``buffer[-a]``
which will /not/ generate any output in case of a failing generator `-a`.
]
[heading Complexity]
[:The overall complexity of the optional generator is defined by the
complexity of its embedded generator. The complexity of the optional
generator itself is O(1).]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_optional]
Basic usage of an optional generator:
[reference_karma_optional1]
Usage and result of an empty optional generator:
[reference_karma_optional2]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:and_predicate And-Predicate Generator (`&a`)]
[heading Description]
The and-predicate generator is used to test, whether the embedded generator
succeeds without generating any output. It succeeds if the embedded generator
succeeds.
[heading Header]
// forwards to <boost/spirit/home/karma/operator/and_predicate.hpp>
#include <boost/spirit/include/karma_and_predicate.hpp>
Also, see __include_structure__.
[heading Model of]
[:__unary_generator_concept__]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is not
defined in __unary_generator_concept__.
[table
[[Expression] [Semantics]]
[[`&a`] [The generator `a` is executed for the sole purpose of
testing whether it succeeds. The and-predicate
generator succeeds if its embedded generator
succeeds (unless the underlying output stream
reports an error). The and-predicate never produces
any output.]]
]
The and generator is implemented by redirecting all output produced by its
embedded generator into a discarding device.
[heading Attributes]
See __karma_comp_attr_notation__.
[table
[[Expression] [Attribute]]
[[`&a` (and-predicate, unary `&`)] [`a: A --> &a: A`]]
]
[note The attribute of the and-predicate is not always `unused_type`, which is
different from Qi's and-predicate. This is necessary as the generator the
and predicate is attached to most of the time needs an attribute.
]
[heading Complexity]
[:The overall complexity of the and-predicate generator is defined by the
complexity of its embedded generator. The complexity of the and-predicate
generator itself is O(1).]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_and_predicate]
Basic usage of an and predicate generator:
[reference_karma_and_predicate]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:not_predicate Not-Predicate Generator (`!a`)]
[heading Description]
The not-predicate generator is used to test, whether the embedded generator
fails, without generating any output. It succeeds if the embedded generator
fails.
[heading Header]
// forwards to <boost/spirit/home/karma/operator/not_predicate.hpp>
#include <boost/spirit/include/karma_not_predicate.hpp>
Also, see __include_structure__.
[heading Model of]
[:__unary_generator_concept__]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is not
defined in __unary_generator_concept__.
[table
[[Expression] [Semantics]]
[[`!a`] [The generator `a` is executed for the sole purpose of
testing whether it succeeds. The not-predicate
generator succeeds if its embedded generator
fails (unless the underlying output stream
reports an error). The not-predicate never produces
any output.]]
]
The not generator is implemented by redirecting all output produced by its
embedded generator into a discarding device.
[heading Attributes]
See __karma_comp_attr_notation__.
[table
[[Expression] [Attribute]]
[[`!a` (not-predicate, unary `!`)] [`a: A --> !a: A`]]
]
[note The attribute of the not-predicate is not always `unused_type`, which is
different from Qi's not-predicate. This is necessary as the generator the
and-predicate is attached to most of the time needs an attribute.
]
[heading Complexity]
[:The overall complexity of the not-predicate generator is defined by the
complexity of its embedded generator. The complexity of the not-predicate
generator itself is O(1).]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_not_predicate]
Basic usage of a not predicate generator:
[reference_karma_not_predicate]
[endsect]
[endsect]

View File

@@ -0,0 +1,687 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
This quick reference section is provided for convenience. You can use
this section as sort of a "cheat-sheet" on the most commonly used Karma
components. It is not intended to be complete, but should give you an
easy way to recall a particular component without having to dig up on
pages upon pages of reference documentation.
[/////////////////////////////////////////////////////////////////////////////]
[section Common Notation]
[variablelist Notation
[[`G`] [Generator type]]
[[`g, a, b, c, d`] [Generator objects]]
[[`A, B, C, D`] [Attribute types of generators `a`, `b`, `c`, and `d`]]
[[`I`] [The iterator type used for generation]]
[[`Unused`] [An `unused_type`]]
[[`Context`] [The enclosing rule's `Context` type]]
[[`attrib`] [An attribute value]]
[[`Attrib`] [An attribute type]]
[[`b`] [A boolean expression]]
[[`B`] [A type to be interpreted in boolean expressions]]
[[`fg`] [A (lazy generator) function with signature `G(Unused, Context)`]]
[[`fa`] [A (semantic action) function with signature `void(Attrib&, Context, bool&)`.
The third parameter is a boolean flag that can be set to false to
force the generator to fail. Both `Context` and the boolean flag are
optional.]]
[[`outiter`] [An output iterator to receive the generated output]]
[[`Ch`] [Character-class specific character type (See __char_class_types__)]]
[[`ch, ch2`] [Character-class specific character (See __char_class_types__)]]
[[`charset`] [Character-set specifier string (example: `"a-z0-9"`)]]
[[`str`] [Character-class specific string (See __char_class_types__)]]
[[`Str`] [Attribute of `str`: `std::basic_string<T>` where `T` is the underlying character type of `str`]]
[[`num`] [Numeric literal, any integer or real number type]]
[[`Num`] [Attribute of `num`: any integer or real number type]]
[[`tuple<>`] [Used as a placeholder for a fusion sequence]]
[[`vector<>`] [Used as a placeholder for an STL container]]
[[`variant<>`] [Used as a placeholder for a boost::variant]]
[[`optional<>`] [Used as a placeholder for a boost::optional]]
]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:primitive_generators Karma Generators]
[section:char Character Generators]
See here for more information about __karma_char__.
[table
[[Expression] [Attribute] [Description]]
[[[karma_char `ch`]] [`Unused`] [Generate `ch`]]
[[[karma_char `lit(ch)`]] [`Unused`] [Generate `ch`]]
[[[karma_char `char_`]] [`Ch`] [Generate character supplied as the attribute]]
[[[karma_char `char_(ch)`]] [`Ch`] [Generate `ch`,
if an attribute is supplied it must match]]
[[[karma_char `char_("c")`]] [`Ch`] [Generate a single char string literal, `c`,
if an attribute is supplied it must match]]
[[[karma_char `char_(ch, ch2)`]][`Ch`] [Generate the character supplied as the attribute,
if it belongs to the character range from
`ch` to `ch2`]]
[[[karma_char `char_(charset)`]][`Ch`] [Generate the character supplied as the attribute,
if it belongs to the character set `charset`]]
[[[karma_char_class `alnum`]] [`Ch`] [Generate the character supplied as the attribute
if it satisfies the concept of `std::isalnum` in
the character set defined by `NS`]]
[[[karma_char_class `alpha`]] [`Ch`] [Generate the character supplied as the attribute
if it satisfies the concept of `std::isalpha` in
the character set defined by `NS`]]
[[[karma_char_class `blank`]] [`Ch`] [Generate the character supplied as the attribute
if it satisfies the concept of `std::isblank` in
the character set defined by `NS`]]
[[[karma_char_class `cntrl`]] [`Ch`] [Generate the character supplied as the attribute
if it satisfies the concept of `std::iscntrl` in
the character set defined by `NS`]]
[[[karma_char_class `digit`]] [`Ch`] [Generate the character supplied as the attribute
if it satisfies the concept of `std::isdigit` in
the character set defined by `NS`]]
[[[karma_char_class `graph`]] [`Ch`] [Generate the character supplied as the attribute
if it satisfies the concept of `std::isgraph` in
the character set defined by `NS`]]
[[[karma_char_class `print`]] [`Ch`] [Generate the character supplied as the attribute
if it satisfies the concept of `std::isprint` in
the character set defined by `NS`]]
[[[karma_char_class `punct`]] [`Ch`] [Generate the character supplied as the attribute
if it satisfies the concept of `std::ispunct` in
the character set defined by `NS`]]
[[[karma_char_class `space`]] [`Ch`] [Generate the character supplied as the attribute
if it satisfies the concept of `std::isspace`, or
a single space character in the character set
defined by `NS`]]
[[[karma_char_class `xdigit`]] [`Ch`] [Generate the character supplied as the attribute
if it satisfies the concept of `std::isxdigit` in
the character set defined by `NS`]]
[[[karma_char_class `lower`]] [`Ch`] [Generate the character supplied as the attribute
if it satisfies the concept of `std::islower` in
the character set defined by `NS`]]
[[[karma_char_class `upper`]] [`Ch`] [Generate the character supplied as the attribute
if it satisfies the concept of `std::isupper` in
the character set defined by `NS`]]
]
[endsect]
[section:string String Generators]
See here for more information about [karma_string String Generators].
[table
[[Expression] [Attribute] [Description]]
[[[karma_string `str`]] [`Unused`] [Generate `str`]]
[[[karma_string `lit(str)`]] [`Unused`] [Generate `str`]]
[[[karma_string `string`]] [`Str`] [Generate string supplied as the attribute]]
[[[karma_string `string(str)`]] [`Str`] [Generate `str`,
if an attribute is supplied it must match]]
[[[karma_symbols `symbols<Attr, T>`]][`Attr`] [Declare a symbol table, `sym`. `Attr` is the
The type of the original attribute to be used
as the key into the symbol generator.
`T` is the data type associated with each key.]]
[[
``
sym.add
(attr1, val1)
(attr2, val2)
/*...more...*/
;
``
]
[N/A] [Add symbols into a symbol table, `sym`.
val1 and val2 are optional data of type `T`,
the data type associated with each key.]]
[[`sym`] [`T`] [Emits entries in the symbol table, `sym`. If
attribute is found in the symbol table,
the corresponding value is emitted. If `sym`
does not store values, the original attribute
is emitted.]]
]
[endsect]
[section:real_number Real Number Generators]
See here for more information about __karma_numeric__.
[table
[[Expression] [Attribute] [Description]]
[[[real_number `lit(num)`]] [`Unused`] [Generate `num`]]
[[[real_number `float_`]] [`float`] [Generate a real number from a `float`]]
[[[real_number `float_(num)`]] [`float`] [Generate `num` as a real number from a `float`,
if an attribute is supplied it must match `num`]]
[[[real_number `double_`]] [`double`] [Generate a real number from a `double`]]
[[[real_number `double_(num)`]] [`double`] [Generate a `num` as a real number from a `double`,
if an attribute is supplied it must match `num`]]
[[[real_number `long_double`]] [`long double`] [Generate a real number from a `long double`]]
[[[real_number `long_double(num)`]] [`long double`] [Generate `num` as a real number from a `long double`,
if an attribute is supplied it must match `num`]]
[[[real_number ``real_generator<
Num, Policies
>()``]]
[`Num`] [Generate a real number `Num` using
the supplied real number formatting policies]]
[[[real_number ``real_generator<
Num, Policies
>()(num)``]]
[`Num`] [Generate real number `num` as a `Num`
using the supplied real number formatting
policies, if an attribute is supplied it must
match]]
]
[endsect]
[section:signed_int Integer Generators]
[table
[[Expression] [Attribute] [Description]]
[[[signed_int `lit(num)`]] [`Unused`] [Generate `num`]]
[[[signed_int `short_`]] [`short`] [Generate a short integer]]
[[[signed_int `short_(num)`]] [`short`] [Generate `num` as a short integer,
if an attribute is supplied it must match]]
[[[signed_int `int_`]] [`int`] [Generate an int]]
[[[signed_int `int_(num)`]] [`int`] [Generate `num` as an int,
if an attribute is supplied it must match]]
[[[signed_int `long_`]] [`long`] [Generate a long integer]]
[[[signed_int `long_(num)`]] [`long`] [Generate `num` as long integer,
if an attribute is supplied it must match]]
[[[signed_int `long_long`]] [`long long`] [Generate a long long]]
[[[signed_int `long_long(num)`]] [`long long`] [Generate `num` as a long long,
if an attribute is supplied it must match]]
[[[signed_int ``int_generator<
Num, Radix, force_sign
>()``]]
[`Num`] [Generate a `Num`]]
[[[signed_int ``int_generator<
Num, Radix, force_sign
>()(num)``]]
[`Num`] [Generate a `num` as a `Num`,
if an attribute is supplied it must match]]
]
[endsect]
[section:unsigned_int Unsigned Integer Generators]
[table
[[Expression] [Attribute] [Description]]
[[[unsigned_int `lit(num)`]] [`Unused`] [Generate `num`]]
[[[unsigned_int `ushort_`]] [`unsigned short`] [Generate an unsigned short integer]]
[[[unsigned_int `ushort_(num)`]] [`unsigned short`] [Generate `num` as an unsigned short integer,
if an attribute is supplied it must match]]
[[[unsigned_int `uint_`]] [`unsigned int`] [Generate an unsigned int]]
[[[unsigned_int `uint_(num)`]] [`unsigned int`] [Generate `num` as an unsigned int,
if an attribute is supplied it must match]]
[[[unsigned_int `ulong_`]] [`unsigned long`] [Generate an unsigned long integer]]
[[[unsigned_int `ulong_(num)`]] [`unsigned long`] [Generate `num` as an unsigned long integer,
if an attribute is supplied it must match]]
[[[unsigned_int `ulong_long`]] [`unsigned long long`] [Generate an unsigned long long]]
[[[unsigned_int `ulong_long(num)`]] [`unsigned long long`] [Generate `num` as an unsigned long long,
if an attribute is supplied it must match]]
[[[unsigned_int `bin`]] [`unsigned int`] [Generate a binary integer from an `unsigned int`]]
[[[unsigned_int `oct`]] [`unsigned int`] [Generate an octal integer from an `unsigned int`]]
[[[unsigned_int `hex`]] [`unsigned int`] [Generate a hexadecimal integer from an `unsigned int`]]
[[[unsigned_int ``uint_generator<
Num, Radix
>()``]]
[`Num`] [Generate an unsigned `Num`]]
[[[unsigned_int ``uint_generator<
Num, Radix
>()(num)``]]
[`Num`] [Generate an unsigned `num` as a `Num`,
if an attribute is supplied it must match]]
]
[endsect]
[section:boolean Boolean Generators]
[table
[[Expression] [Attribute] [Description]]
[[[boolean `lit(num)`]] [`Unused`] [Generate `num`]]
[[[boolean `bool_`]] [`bool`] [Generate a boolean]]
[[[boolean `bool_(b)`]] [`bool`] [Generate `b` as a boolean,
if an attribute is supplied it must match]]
[[[boolean ``bool_generator<
B, Policies
>()``]]
[`B`] [Generate a boolean of type `B`]]
[[[boolean ``bool_generator<
B, Policies
>()(b)``]]
[`B`] [Generate a boolean `b` as a `B`,
if an attribute is supplied it must match]]
]
[endsect]
[section:stream Stream Generators]
See here for more information about [karma_stream Stream Generators].
[table
[[Expression] [Attribute] [Description]]
[[[karma_stream `stream`]] [`hold_any`] [Generate narrow character (`char`) based output
using the matching streaming `operator<<()`]]
[[[karma_stream `stream(s)`]] [`Unused`] [Generate narrow character (`char`) based output
from the immediate argument `s` using the matching
streaming `operator<<()`]]
[[[karma_stream `wstream`]] [`whold_any`] [Generate wide character (`wchar_t`) based output
using the matching streaming `operator<<()`]]
[[[karma_stream `wstream(s)`]] [`Unused`] [Generate wide character (`wchar_t`) based output
from the immediate argument `s` using the matching
streaming `operator<<()`]]
[
[[karma_stream ``stream_generator<
Char
>()``]] [`basic_hold_any<Char>`] [Generate output based on the given character type
(`Char`) using the matching streaming `operator<<()`]]
[
[[karma_stream ``stream_generator<
Char
>()(s)``]] [`Unused`] [Generate output based on the given character type
`Char` from the immediate argument `s` using the
matching streaming `operator<<()`]]
]
[endsect]
[section:binary Binary Generators]
See here for more information about __karma_binary__.
[table
[[Expression] [Attribute] [Description]]
[[[karma_native_binary `byte_`]] [8 bits native endian] [Generate an 8 bit binary]]
[[[karma_native_binary `word`]] [16 bits native endian] [Generate a 16 bit binary in native endian representation]]
[[[karma_big_binary `big_word`]] [16 bits big endian] [Generate a 16 bit binary in big endian representation]]
[[[karma_little_binary `little_word`]] [16 bits little endian] [Generate a 16 bit binary in little endian representation]]
[[[karma_native_binary `dword`]] [32 bits native endian] [Generate a 32 bit binary in native endian representation]]
[[[karma_big_binary `big_dword`]] [32 bits big endian] [Generate a 32 bit binary in big endian representation]]
[[[karma_little_binary `little_dword`]][32 bits little endian] [Generate a 32 bit binary in little endian representation]]
[[[karma_native_binary `qword`]] [64 bits native endian] [Generate a 64 bit binary in native endian representation]]
[[[karma_big_binary `big_qword`]] [64 bits big endian] [Generate a 64 bit binary in big endian representation]]
[[[karma_little_binary `little_qword`]][64 bits little endian] [Generate a 64 bit binary in little endian representation]]
[[`pad(num)`] [`Unused`] [Generate additional null bytes allowing to align generated
output with memory addresses divisible by `num`.]]
]
[endsect]
[section:auxiliary Auxiliary Generators]
See here for more information about __karma_auxiliary__.
[table
[[Expression] [Attribute] [Description]]
[[[karma_attr_cast `attr_cast<Exposed>(a)`]] [`Exposed`] [Invoke `a` while supplying an attribute of type `Exposed`.]]
[[__karma_eol__] [`Unused`] [Generate the end of line (`\n`)]]
[[__karma_eps__] [`Unused`] [Generate an empty string]]
[[__karma_feps__] [`Unused`] [If `b` is true, generate an empty string]]
[[[karma_lazy `lazy(fg)`]]
[Attribute of `G` where `G`
is the return type of `fg`] [Invoke `fg` at generation time, returning a generator
`g` which is then called to generate.]]
[[[karma_lazy `fg`]] [see [karma_lazy `lazy(fg)`] above] [Equivalent to [karma_lazy `lazy(fg)`]]]
]
[endsect]
[section:auto Auto Generators]
See here for more information about [karma_auto Auto Generators].
[table
[[Expression] [Attribute] [Description]]
[[[karma_auto `auto_`]] [`hold_any`] [Generate output using a generator
created from the supplied attribute type
using the __create_generator__ API function.]]
]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:operators Generator Operators]
See here for more information about __karma_operator__.
[table
[[Expression] [Attribute] [Description]]
[[[link spirit.karma.reference.operator.not_predicate `!a`]]
[`A`] [Not predicate. Ensure that `a` does not succeed
generating, but don't create any output]]
[[[link spirit.karma.reference.operator.and_predicate `&a`]]
[`A`] [And predicate. Ensure that `a` does succeed
generating, but don't create any output]]
[[[link spirit.karma.reference.operator.optional `-a`]]
[`optional<A>`] [Optional. Generate `a` zero or one time]]
[[[link spirit.karma.reference.operator.kleene `*a`]]
[`vector<A>`] [Kleene. Generate `a` zero or more times]]
[[[link spirit.karma.reference.operator.plus `+a`]]
[`vector<A>`] [Plus. Generate `a` one or more times]]
[[[link spirit.karma.reference.operator.alternative `a | b`]]
[`variant<A, B>`] [Alternative. Generate `a` or `b`]]
[[[link spirit.karma.reference.operator.sequence `a << b`]]
[`tuple<A, B>`] [Sequence. Generate `a` followed by `b`]]
[[[link spirit.karma.reference.operator.list `a % b`]]
[`vector<A>`] [List. Generate `a` delimited `b` one or more times]]
]
[:For more information about the attribute propagation rules implemented by the
compound generators please see __sec_karma_compound__.]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:directives Generator Directives]
See here for more information about __karma_directive__.
[table
[[Expression] [Attribute] [Description]]
[[[karma_upperlower `lower`]`[a]`] [`A`] [Generate `a` as lower case]]
[[[karma_upperlower `upper`]`[a]`] [`A`] [Generate `a` as upper case]]
[[[karma_align `left_align`]`[a]`] [`A`] [Generate `a` left aligned in column of width
`BOOST_KARMA_DEFAULT_FIELD_LENGTH`]]
[[[karma_align `left_align`]`(num)[a]`] [`A`] [Generate `a` left aligned in column of width `num`]]
[[[karma_align `left_align`]`(g)[a]`] [`A`] [Generate `a` left aligned in column of width
`BOOST_KARMA_DEFAULT_FIELD_LENGTH` while using `g` to
generate the necessary padding]]
[[[karma_align `left_align`]`(num, g)[a]`][`A`] [Generate `a` left aligned in column of width `num`
while using `g` to generate the necessary
padding]]
[[[karma_align `center`]`[a]`] [`A`] [Generate `a` centered in column of width
`BOOST_KARMA_DEFAULT_FIELD_LENGTH`]]
[[[karma_align `center`]`(num)[a]`] [`A`] [Generate `a` centered in column of width `num`]]
[[[karma_align `center`]`(g)[a]`] [`A`] [Generate `a` centered in column of width
`BOOST_KARMA_DEFAULT_FIELD_LENGTH` while using `g` to
generate the necessary padding]]
[[[karma_align `center`]`(num, g)[a]`] [`A`] [Generate `a` centered in column of width `num`
while using `g` to generate the necessary
padding]]
[[[karma_align `right_align`]`[a]`] [`A`] [Generate `a` right aligned in column of width
`BOOST_KARMA_DEFAULT_FIELD_LENGTH`]]
[[[karma_align `right_align`]`(num)[a]`] [`A`] [Generate `a` right aligned in column of width `num`]]
[[[karma_align `right_align`]`(g)[a]`] [`A`] [Generate `a` right aligned in column of width
`BOOST_KARMA_DEFAULT_FIELD_LENGTH` while using `g` to
generate the necessary padding]]
[[[karma_align `right_align`]`(num, g)[a]`][`A`][Generate `a` right aligned in column of width `num`
while using `g` to generate the necessary
padding]]
[[[karma_maxwidth `maxwidth`]`[a]`] [`A`] [Generate `a` truncated to column of width
`BOOST_KARMA_DEFAULT_FIELD_MAXWIDTH`]]
[[[karma_maxwidth `maxwidth`]`(num)[a]`] [`A`] [Generate `a` truncated to column of width `num`]]
[[[karma_repeat `repeat`]`[a]`] [`vector<A>`] [Repeat `a` zero or more times]]
[[[karma_repeat `repeat`]`(num)[a]`] [`vector<A>`] [Repeat `a` `num` times]]
[[[karma_repeat `repeat`]`(num1, num2)[a]`] [`vector<A>`] [Repeat `a` `num1` to `num2` times]]
[[[karma_repeat `repeat`]`(num, inf)[a]`] [`vector<A>`] [Repeat `a` `num` or more times]]
[[__karma_verbatim__`[a]`][`A`] [Disable delimited generation for `a`. Performs post delimiting.]]
[[[karma_delimit `delimit`]`[a]`] [`A`] [Reestablish the delimiter that got inhibited by verbatim]]
[[[karma_delimit `delimit`]`(d)[a]`] [`A`] [Use `d` as a delimiter for generating `a`]]
[[[karma_no_delimit `no_delimit`]`[a]`] [`A`] [Disable delimited generation for `a`. No post-delimiting step performed.]]
[[__karma_as__`()[a]`] [`A`] [Force atomic output from arbitrary attribute types]]
[[__karma_as_string__`[a]`] [`A`] [Force atomic output from string attributes]]
[[__karma_as_wstring__`[a]`] [`A`] [Force atomic output from wide character string attributes]]
[[__karma_omit__`[a]`] [`A`] [Consume the attribute type of `a` without generating anything.
The embedded generator will be always executed.]]
[[__karma_skip__`[a]`] [`A`] [Consume the attribute type of `a` without generating anything.
The embedded generator will never be executed.]]
[[__karma_duplicate__`[a]`] [`A`] [The supplied attribute will be duplicated and passed unchanged to
all embedded elements of a sequence.]]
[[__karma_buffer__`[a]`][`A`] [Temporarily intercept the output generated by `a`,
flushing it only after `a` succeeded]]
[[[karma_columns `columns`]`[a]`] [`A`] [Generate `a` split into
`BOOST_KARMA_DEFAULT_COLUMNS` number of columns using
`karma::eol` as column delimiter]]
[[[karma_columns `columns`]`(num)[a]`] [`A`] [Generate `a` split into
`num` number of columns using
`karma::eol` as column delimiter]]
[[[karma_columns `columns`]`(g)[a]`] [`A`] [Generate `a` split into
`BOOST_KARMA_DEFAULT_COLUMNS` number of columns using
`g` as column delimiter]]
[[[karma_columns `columns`]`(num, g)[a]`][`A`][Generate `a` split into
`num` number of columns using
`g` as column delimiter]]
]
[endsect]
[section:action Generator Semantic Actions]
[table
[[Expression] [Attribute] [Description]]
[[`g[fa]`] [Attribute of `g`] [Call semantic action `fa` before invoking `g`]]
]
[endsect]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section Compound Attribute Rules]
[heading Notation]
The notation we use is of the form:
a: A, b: B, ... --> composite-expression: composite-attribute
`a`, `b`, etc. are the operands. `A`, `B`, etc. are the operand's
attribute types. `composite-expression` is the expression involving the
operands and `composite-attribute` is the resulting attribute type of
the composite expression.
For instance:
a: A, b: B --> (a << b): tuple<A, B>
which reads as: given, `a` and `b` are generators, and `A` is the type
of the attribute of `a`, and `B` is the type of the attribute of `b`, then the
type of the attribute of `a << b` will be `tuple<A, B>`.
[important In the attribute tables, we will use `vector<A>` and
`tuple<A, B...>` as placeholders only. The notation of `vector<A>`
stands for ['any __stl__ container] holding elements of type `A` and the
notation `tuple<A, B...>` stands for ['any __fusion__ sequence] holding
`A`, `B`, ... etc. elements. The notation of `variant<A, B, ...>` stands for
['a __boost_variant__] capable of holding `A`, `B`, ... etc. elements. Finally,
`Unused` stands for __unused_type__. ]
[heading Compound Generator Attribute Types]
[table
[[Expression] [Attribute]]
[[__karma_sequence__ (`a << b`)]
[``a: A, b: B --> (a << b): tuple<A, B>
a: A, b: Unused --> (a << b): A
a: Unused, b: B --> (a << b): B
a: Unused, b: Unused --> (a << b): Unused
a: A, b: A --> (a << b): vector<A>
a: vector<A>, b: A --> (a << b): vector<A>
a: A, b: vector<A> --> (a << b): vector<A>
a: vector<A>, b: vector<A> --> (a << b): vector<A>``]]
[[__karma_alternative__ (`a | b`)]
[``a: A, b: B --> (a | b): variant<A, B>
a: A, b: Unused --> (a | b): A
a: Unused, b: B --> (a | b): B
a: Unused, b: Unused --> (a | b): Unused
a: A, b: A --> (a | b): A``]]
[[[karma_kleene Kleene (`*a`)]]
[``a: A --> *a: vector<A>
a: Unused --> *a: Unused``]]
[[__karma_plus__ (`+a`)]
[``a: A --> +a: vector<A>
a: Unused --> +a: Unused``]]
[[__karma_list__ (`a % b`)]
[``a: A, b: B --> (a % b): vector<A>
a: Unused, b: B --> (a % b): Unused``]]
[[[karma_repeat Repetition] (`repeat[]`)]
[``a: A --> repeat(...,...)[a]: vector<A>
a: Unused --> repeat(...,...)[a]: Unused``]]
[[__karma_optional__ (`-a`)]
[``a: A --> -a: optional<A>
a: Unused --> -a: Unused``]]
[[__karma_and_predicate__ (`&a`)] [`a: A --> &a: A`]]
[[__karma_not_predicate__ (`!a`)] [`a: A --> !a: A`]]
]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:non_terminals Nonterminals]
See here for more information about __karma_nonterminal__.
[variablelist Notation
[[`RT`] [Synthesized attribute. The rule or grammar's return type.]]
[[`Arg1`, `Arg2`, `ArgN`] [Inherited attributes. Zero or more arguments.]]
[[`L1`, `L2`, `LN`] [Zero or more local variables.]]
[[`r, r2`] [Rules]]
[[`g`] [A grammar]]
[[`p`] [A generator expression]]
[[`my_grammar`] [A user defined grammar]]
]
[variablelist Terminology
[[Signature] [`RT(Arg1, Arg2, ... ,ArgN)`. The signature specifies
the synthesized (return value) and inherited (arguments)
attributes.]]
[[Locals] [`locals<L1, L2, ..., LN>`. The local variables.]]
[[Delimiter] [The delimit-generator type]]
]
[variablelist Template Arguments
[[`Iterator`] [The iterator type you will use for parsing.]]
[[`A1`, `A2`, `A3`] [Can be one of 1) Signature 2) Locals 3) Delimiter.]]
]
[table
[[Expression] [Description]]
[[`rule<OutputIterator, A1, A2, A3> r(name);`] [Rule declaration. `OutputIterator` is required.
`A1, A2, A3` are optional and can be specified in any order.
`name` is an optional string that gives the rule
its name, useful for debugging.]]
[[`rule<OutputIterator, A1, A2, A3> r(r2);`] [Copy construct rule `r` from rule `r2`.]]
[[`r = r2;`] [Assign rule `r2` to `r`. `boost::shared_ptr` semantics.]]
[[`r.alias()`] [Return an alias of `r`. The alias is a generator that
holds a reference to `r`. Reference semantics.]]
[[`r.copy()`] [Get a copy of `r`.]]
[[`r.name(name)`] [Set the name of a rule]]
[[`r.name()`] [Get the name of a rule]]
[[debug(r)] [Debug rule `r`]]
[[`r = g;`] [Rule definition]]
[[`r %= g;`] [Auto-rule definition. The attribute of `g` should be
compatible with the synthesized attribute of `r`. When `g`
is successful, its attribute is automatically propagated
to `r`'s synthesized attribute.]]
[[
``
template <typename OutputIterator>
struct my_grammar : grammar<OutputIterator, A1, A2, A3>
{
my_grammar() : my_grammar::base_type(start, name)
{
// Rule definitions
start = /* ... */;
}
rule<OutputIterator, A1, A2, A3> start;
// more rule declarations...
};
``
] [Grammar definition. `name` is an optional string that gives the
grammar its name, useful for debugging.]]
[[my_grammar<OutputIterator> g] [Instantiate a grammar]]
[[`g.name(name)`] [Set the name of a grammar]]
[[`g.name()`] [Get the name of a grammar]]
]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:semantic_actions Generator Semantic Actions]
Semantic Actions may be attached to any generator as follows:
g[f]
where `f` is a function with the signatures:
void f(Attrib&);
void f(Attrib&, Context&);
void f(Attrib&, Context&, bool&);
You can use __boost_bind__ to bind member functions. For function
objects, the allowed signatures are:
void operator()(Attrib&, unused_type, unused_type) const;
void operator()(Attrib&, Context&, unused_type) const;
void operator()(Attrib&, Context&, bool&) const;
The `unused_type` is used in the signatures above to signify 'don't
care'.
For more information see __karma_actions__.
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section Phoenix]
__phoenix__ makes it easier to attach semantic actions. You just
inline your lambda expressions:
g[phoenix-lambda-expression]
__karma__ provides some __phoenix__ placeholders to access important
information from the `Attrib` and `Context` that are otherwise fiddly to extract.
[variablelist Spirit.Karma specific Phoenix placeholders
[[`_1, _2, ... , _N`] [Nth attribute of `g`]]
[[`_val`] [The enclosing rule's synthesized attribute.]]
[[`_r1, _r2, ... , _rN`] [The enclosing rule's Nth inherited attribute.]]
[[`_a, _b, ... , _j`] [The enclosing rule's local variables (`_a` refers to the first).]]
[[`_pass`] [Assign `false` to `_pass` to force a generator failure.]]
]
[important All placeholders mentioned above are defined in the namespace
`boost::spirit` and, for your convenience, are available in the
namespace `boost::spirit::karma` as well.]
For more information see __karma_actions__.
[endsect]

View File

@@ -0,0 +1,223 @@
[/==============================================================================
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2001-2011 Joel de Guzman
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:stream Stream Generators]
This module includes the description of the different variants of the `stream`
generator. It can be used to utilize existing streaming operators
(`operator<<(std::ostream&, ...)`) for output generation.
[heading Header]
// forwards to <boost/spirit/home/karma/stream.hpp>
#include <boost/spirit/include/karma_stream.hpp>
Also, see __include_structure__.
[section:stream Stream Generators (`stream`, `wstream`, etc.)]
[heading Description]
The `stream_generator` is a primitive which allows to use pre-existing standard
streaming operators for output generation integrated with __karma__. It
provides a wrapper generator dispatching the value to output to the stream
operator of the corresponding type. Any value `a` to be formatted using the
`stream_generator` will result in invoking the standard streaming operator
for its type `A`, for instance:
std::ostream& operator<< (std::ostream&, A const&);
[heading Header]
// forwards to <boost/spirit/home/karma/stream.hpp>
#include <boost/spirit/include/karma_stream.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::stream // alias: boost::spirit::karma::stream`]]
[[`boost::spirit::wstream // alias: boost::spirit::karma::wstream`]]
]
[heading Synopsis]
template <typename Char>
struct stream_generator;
[heading Template parameters]
[table
[[Parameter] [Description] [Default]]
[[`Char`] [The character type to use to generate
the output. This type will be used while
assigning the generated characters to the
underlying output iterator.] [`char`]]
]
[heading Model of]
[:__primitive_generator_concept__]
[variablelist Notation
[[`s`] [A variable instance of any type with a defined matching
streaming `operator<<()` or a __karma_lazy_argument__ that
evaluates to any type with a defined matching streaming
`operator<<()`.]]
]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __primitive_generator_concept__.
[table
[[Expression] [Description]]
[[`stream`] [Call the streaming `operator<<()` for the type
of the mandatory attribute. The output emitted
by this operator will be the result of the
`stream` generator. This generator never fails
(unless the underlying output stream reports an
error). The character type of the I/O ostream
is assumed to be `char`.]]
[[`stream(s)`] [Call the streaming `operator<<()` for the type
of the immediate value `s`. The output emitted
by this operator will be the result of the
`stream` generator. This generator never fails
(unless the underlying output stream reports an
error). The character type of the I/O ostream
is assumed to be `char`.]]
[[`wstream`] [Call the streaming `operator<<()` for the type
of the mandatory attribute. The output emitted
by this operator will be the result of the
`stream` generator. This generator never fails
(unless the underlying output stream reports an
error). The character type of the I/O ostream
is assumed to be `wchar_t`.]]
[[`wstream(s)`] [Call the streaming `operator<<()` for the type
of the immediate value `s`. The output emitted
by this operator will be the result of the
`stream` generator. This generator never fails
(unless the underlying output stream reports an
error). The character type of the I/O ostream
is assumed to be `wchar_t`.]]
]
All generators listed in the table above are predefined specializations of the
`stream_generator<Char>` basic stream generator type described below. It is
possible to directly use this type to create stream generators using an
arbitrary underlying character type.
[table
[[Expression] [Semantics]]
[
[``stream_generator<
Char
>()``] [Call the streaming `operator<<()` for the type
of the mandatory attribute. The output emitted
by this operator will be the result of the
`stream` generator. This generator never fails
(unless the underlying output stream reports an
error). The character type of the I/O ostream
is assumed to be `Char`]]
[
[``stream_generator<
Char
>()(s)``] [Call the streaming `operator<<()` for the type
of the immediate value `s`. The output emitted
by this operator will be the result of the
`stream` generator. This generator never fails
(unless the underlying output stream reports an
error). The character type of the I/O ostream
is assumed to be `Char`.]]
]
[heading Additional Requirements]
All of the stream generators listed above require the type of the value to
generate output for (either the immediate value or the associated attribute) to
implement a streaming operator conforming to the usual I/O streams conventions
(where `attribute_type` is the type of the value to generate output for):
template <typename Ostream>
Ostream& operator<< (Ostream& os, attribute_type const& attr)
{
// type specific output generation
return os;
}
This operator will be called by the stream generators to gather the output for
the attribute of type `attribute_type`. All data streamed into the given
`Ostream` will end up being generated by the corresponding stream generator
instance.
[note If the `stream` generator is invoked inside a [karma_format `format`]
(or [karma_format `format_delimited`]) stream manipulator the `Ostream`
passed to the `operator<<()` will have registered (imbued) the same
standard locale instance as the stream the [karma_format `format`] (or
[karma_format `format_delimited`]) manipulator has been used with.
This ensures all facets registered (imbued) with the original I/O
stream object are used during output generation.
]
[heading Attributes]
[table
[[Expression] [Attribute]]
[[`stream`] [`hold_any`, attribute is mandatory (otherwise compilation will fail)]]
[[`stream(s)`] [__unused__]]
[[`wstream`] [`whold_any`, attribute is mandatory (otherwise compilation will fail)]]
[[`wstream(s)`] [__unused__]]
[[`stream_generator<Char>()`] [`basic_hold_any<Char>`, attribute is mandatory (otherwise compilation will fail)]]
[[`stream_generator<Char>()(s)`] [__unused__]]
]
[important The attribute type `hold_any` exposed by some of the stream
generators is semantically and syntactically equivalent to
the type implemented by __boost_any__. It has been added to /Spirit/
as it has better a performance and a smaller footprint if compared to
__boost_any__.
]
[note In addition to their usual attribute of type `Attrib` all listed generators
accept an instance of a `boost::optional<Attrib>` as well. If the
`boost::optional<>` is initialized (holds a value) the generators behave
as if their attribute was an instance of `Attrib` and emit the value stored
in the `boost::optional<>`. Otherwise the generators will fail.]
[heading Complexity]
[:O(N), where N is the number of characters emitted by the stream generator]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_stream]
And a class definition used in the examples:
[reference_karma_complex]
[reference_karma_stream_complex]
Basic usage of `stream` generators:
[reference_karma_stream]
[endsect]
[endsect]

View File

@@ -0,0 +1,358 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:string String Generators]
This module includes different string oriented generators allowing to output
character sequences. It includes the `symbols` generator and variants of the
`string` generator.
[heading Module Header]
// forwards to <boost/spirit/home/karma/string.hpp>
#include <boost/spirit/include/karma_string.hpp>
Also, see __include_structure__.
[section:string String Generators (`string`, `lit`)]
[heading Description]
The string generators described in this section are:
The `string` generator emits a string of characters. The `string` generator
is implicitly verbatim: the `delimit` parser is not applied in between
characters of the string. The `string` generator has an associated
__karma_char_encoding_namespace__. This is needed when doing basic operations
such as forcing lower or upper case. Examples:
string("Hello")
string(L"Hello")
string(s) // s is a std::string
`lit`, like `string`, also emits a string of characters. The main
difference is that `lit` does not consumes an attribute. A plain
string like `"hello"` or a `std::basic_string` is equivalent to a `lit`.
Examples:
"Hello"
lit("Hello")
lit(L"Hello")
lit(s) // s is a std::string
[heading Header]
// forwards to <boost/spirit/home/karma/string/lit.hpp>
#include <boost/spirit/include/karma_string.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::lit // alias: boost::spirit::karma::lit`]]
[[`ns::string`]]
]
In the table above, `ns` represents a __karma_char_encoding_namespace__ used by the
corresponding string generator.
[heading Model of]
[:__primitive_generator_concept__]
[variablelist Notation
[[`s`] [Character-class specific string (See __char_class_types__),
or a __karma_lazy_argument__ that evaluates to a
character-class specific string value]]
[[`S`] [The type of a character-class specific string `s`.]]
[[`ns`] [A __karma_char_encoding_namespace__.]]]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is
not defined in __primitive_generator_concept__.
[table
[[Expression] [Description]]
[[`s`] [Generate the string literal `s`. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`lit(s)`] [Generate the string literal `s`. This generator
never fails (unless the underlying output stream
reports an error).]]
[[`ns::string`] [Generate the string provided by a mandatory
attribute interpreted in the character set defined
by `ns`. This generator never fails (unless the
underlying output stream reports an error).]]
[[`ns::string(s)`] [Generate the string `s` as provided by the
immediate literal value the generator is initialized
from. If this generator has an associated attribute
it succeeds only if the attribute is equal
to the immediate literal (unless the underlying
output stream reports an error). Otherwise this
generator fails and does not generate any output.]]
]
[note The generators `lit(s)` and `string(s)` can be initialized either
using a string literal value (i.e. `"abc"`), or using a
`std::basic_string<char_type, ...>`, where `char_type` is the required
value type of the underlying character sequence.]
[caution The generator `string(s)` up to version 2.4.1 of Spirit has an
undocumented feature. Given argument `s` generator succeeds as long as
`s` is a prefix of given attribute. This problem has been fixed in
Spirit V2.4.2.]
[heading Attributes]
[table
[[Expression] [Attribute]]
[[`s`] [__unused__]]
[[`lit(s)`] [__unused__]]
[[`ns::string`] [`S`, attribute is mandatory (otherwise compilation
will fail)]]
[[`ns::string(s)`] [`S`, attribute is optional, if it is supplied, the
generator compares the attribute with `s` and
succeeds only if both are equal, failing otherwise]]
]
[note In addition to their usual attribute of type `S` all listed generators
accept an instance of a `boost::optional<S>` as well. If the
`boost::optional<>` is initialized (holds a value) the generators behave
as if their attribute was an instance of `S` and emit the value stored
in the `boost::optional<>`. Otherwise the generators will fail.]
[heading Complexity]
[:O(N), where N is the number of characters emitted by the string generator]
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_string]
Basic usage of `string` generators:
[reference_karma_string]
[endsect]
[/------------------------------------------------------------------------------]
[section:symbols Symbols Generator (`symbols`)]
[heading Description]
The class `symbols` implements an 'inverse' symbol table: an associative
container (or map) of key-value pairs where the values are (most of the time)
strings. It maps the value to be generated (the key) to any other value which
will be emitted instead of the original key.
The Karma symbol table class `symbols` is-a generator, an instance of which may
be used anywhere in the grammar specification. It is an example of a
dynamic generator. A dynamic generator is characterized by its ability to
modify its behavior at run time. Initially, an empty symbols object
will emit nothing. At any time, symbols may be added, thus, dynamically
altering its behavior.
[heading Header]
// forwards to <boost/spirit/home/karma/string/symbols.hpp>
#include <boost/spirit/include/karma_symbols.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::karma::symbols`]]
]
[heading Synopsis]
template <typename Attrib, typename T, typename Lookup
, typename CharEncoding, typename Tag>
struct symbols;
[heading Template parameters]
[table
[[Parameter] [Description] [Default]]
[[`Attrib`] [The type of the original attribute to be used as
the key into the symbol generator (the symbol).] [`char`]]
[[`T`] [The data type associated
with each key.] [__unused_type__]]
[[`Lookup`] [The symbol search implementation]
[if T is `unused_type`, `std::set<Attrib>`,
and `std::map<Attrib, T>` otherwise]]
[[`CharEncoding`] [Used for character set selection, normally not
used by end user.] [__unused_type__]]
[[`Tag`] [Used for character set selection, normally not
used by end user.] [__unused_type__]]
]
[heading Model of]
[:__primitive_generator_concept__]
[variablelist Notation
[[`Sym`] [A `symbols` type.]]
[[`Attrib`] [An attribute type.]]
[[`T`] [A data type.]]
[[`sym`, `sym2`][`symbols` objects.]]
[[`sseq`] [An __stl__ container of strings.]]
[[`dseq`] [An __stl__ container of data with `value_type` `T`.]]
[[`s1`...`sN`] [A __string__.]]
[[`d1`...`dN`] [Objects of type `T`.]]
[[`f`] [A callable function or function object.]]
[[`f`, `l`] [`ForwardIterator` first/last pair.]]
]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is not
defined in __primitive_generator_concept__.
[table
[[Expression] [Semantics]]
[[`Sym()`] [Construct an empty symbols object instance named `"symbols"`.]]
[[`Sym(name)`] [Construct an empty symbols object instance named `name`.]]
[[`Sym(sym2)`] [Copy construct a symbols from `sym2` (Another `symbols` object).]]
[[`Sym(sseq)`] [Construct symbols from `sseq` (An __stl__ container of
symbols of type `Attrib`) named `"symbols"`.]]
[[`Sym(sseq, name)`] [Construct symbols from `sseq` (an __stl__ container of
symbols of type `Attrib`) named `name`.]]
[[`Sym(sseq, dseq)`] [Construct symbols from `sseq` and `dseq`
(An __stl__ container of symbols of type `Attrib` and an
__stl__ container of data with `value_type` `T`)
which is named `"symbols"`.]]
[[`Sym(sseq, dseq, name)`] [Construct symbols from `sseq` and `dseq`
(An __stl__ container of symbols of type `Attrib` and an
__stl__ container of data with `value_type` `T`)
which is named `name`.]]
[[`sym = sym2`] [Assign `sym2` to `sym`.]]
[[`sym = s1, s2, ..., sN`] [Assign one or more symbols (`s1`...`sN`) to `sym`. The
associated data values of type `T` are default constructed.]]
[[`sym += s1, s2, ..., sN`] [Add one or more symbols (`s1`...`sN`) to `sym`. The
associated data values of type `T` are default constructed.]]
[[`sym.add(s1)(s2)...(sN)`] [Add one or more symbols (`s1`...`sN`) to `sym`. The
associated data values of type `T` are default constructed.]]
[[`sym.add(s1, d1)(s2, d2)...(sN, dN)`]
[Add one or more symbols (`s1`...`sN`)
with associated data (`d1`...`dN`) to `sym`.]]
[[`sym -= s1, s2, ..., sN`] [Remove one or more symbols (`s1`...`sN`) from `sym`.]]
[[`sym.remove(s1)(s2)...(sN)`] [Remove one or more symbols (`s1`...`sN`) from `sym`.]]
[[`sym.clear()`] [Erase all of the symbols in `sym`.]]
[[`sym.at(s)`] [Return a reference to the object associated
with symbol, `s`. If `sym` does not already
contain such an object, `at` inserts the default
object `T()`.]]
[[`sym.find(s)`] [Return a pointer to the object associated
with symbol, `s`. If `sym` does not already
contain such an object, `find` returns a null
pointer.]]
[[`sym.for_each(f)`] [For each symbol in `sym` `s` invoke
`f(typename Lookup::value_type)`.]]
[[`sym.name()`] [Retrieve the current name of the symbols object.]]
[[`sym.name(name)`] [Set the current name of the symbols object to be `name`.]]
]
The symbols generator uses the supplied attribute as the key to be looked up
in the internal associative container. If the key exists the generator emits
the associated value and succeeds (unless the underlying output stream reports
an error). If the value type stored in the symbol generator is __unused_type__
it will emit the key instead. If the key does not exist the generator fails
while not emitting anything.
[heading Attributes]
The attribute of `symbol<Attrib, T>` is `Attrib`.
If the supplied attribute is a __fusion__ sequence, then the symbol table
generator will use the first element of that __fusion__ sequence as the key
to be used for lookup. The type of that first element needs to be convertible
to `Attrib`. In this case the second element of the __fusion__ sequence is used
as the attribute while calling a generator derived from the value stored in the
symbol table for the found entry.
If the supplied attribute is a container type (__customize_is_container__
resolves to `mpl::true_`), then the symbol table generator will use the first
element stored in that container as the key to be used for lookup. The
`value_type` (returned by __customize_container_value__) has to be convertible
to `Attrib`. In this case the second element stored in that container is used
as the attribute while calling a generator derived from the value stored in the
symbol table for the found entry.
If the supplied attribute is not a __fusion__ sequence and not a container
type, the supplied attribute is directly used as the key for item lookup. The
attribute is used as the attribute while calling a generator derived from the
value stored in the symbol table for the found entry.
In any case, because the supplied key (i.e. either the first element of the
__fusion__ sequence, the first container element, or the attribute otherwise)
is passed as the attribute to a generator derived from the value
stored in the symbol table for the found entry, the symbol table may store
generators, which will produce output based on that value. For instance:
// The symbol table maps a single character key to a rule<>
// The rule<> exposes an attribute of char as well
rule<output_iterator_type, char()> r1 = char_;
symbols<char, rule<output_iterator_type, char()> > sym;
sym.add
('j', r1.alias())
('h', r1.alias())
('t', r1.alias())
('k', r1.alias())
;
// Supplying a fusion vector as the attribute will use the first element
// (the 'j') as the key to be looked up, while the second element (the 'J')
// is passed on as the attribute to the rule<> stored in the symbol table.
// Consequently, the example generates a single 'J'.
BOOST_ASSERT(test("J", sym, make_vector('j', 'J')));
[heading Complexity]
The default implementation uses a `std::map<>` or a `std::set<>` with a
complexity of:
[:O(log n)]
Where n is the number of stored symbols.
[heading Example]
[note The test harness for the example(s) below is presented in the
__karma_basics_examples__ section.]
Some includes:
[reference_karma_includes]
Some using declarations:
[reference_karma_using_declarations_symbols]
Basic usage of `symbol` generators:
[reference_karma_symbols]
[endsect] [/ symbols]
[endsect]

View File

@@ -0,0 +1,53 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section Quick Start]
[heading Spirit.Karma - what's that?]
Throughout the description of __karma__ we will try to align ourselves very
much with the documentation for __qi__. The reasons are many fold:
* __karma__ is the counterpart to __qi__. Some people say it's the Yin to
__qi__'s Yang. __karma__ is generating byte sequences from internal data
structures as __qi__ is parsing byte sequences into those (very same) internal
data structures.
* Both libraries have an almost identical structure, very similar semantics,
and are both built using identical tools. Both libraries implement a language
casting the specifics of their domain (parsing and generating) into a simple
interface.
Why should you use a generator library for such a simple thing as output
generation? Programmers have been using `printf`, `std::stream` formatting, or
`boost::format` for quite some time. The answer is - yes, for simple output
formatting tasks those familiar tools might be a quick solution. But experience
shows: as soon as the formatting requirements are becoming more complex output
generation is getting more and more challenging in terms of readability,
maintainability, and flexibility of the code. Last, but not least, it turns out
that code using __karma__ runs much faster than equivalent code using either
of the 'straight' methods mentioned above (see here for some numbers:
__sec_karma_numeric_performance__)
You might argue that more complex tasks require more complex tools. But this
turns out not to be the case! The whole Spirit library is designed to be simple
to use, while being scalable from trivial to very complicated applications.
In terms of development simplicity and ease in deployment, the same is true for
__karma__ as has been described elsewhere in this documentation for __qi__: the
entire library consists of only header files, with no libraries to link against
or build. Just put the spirit distribution in your include path, compile and
run. Code size? Very tight, essentially comparable to hand written code.
The __karma__ tutorials are built in a walk through style, starting with
elementary things growing step by step in complexity. And again: keep in mind
output generation is the exact opposite of parsing. Everything you already
learnt about parsing using __qi__ is applicable to generating formatted output
using __karma__. All you have to do is to look at __karma__ as being a
mirror image of __qi__.
[endsect] [/Quick Start]

View File

@@ -0,0 +1,176 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section Warming up]
Learning how to use __karma__ is really simple. We will start from trivial
examples, ramping up as we go.
[heading Trivial Example #1 Generating a number]
Let's create a generator that will output a floating-point number:
double_
Easy huh? The above code actually instantiates a Spirit floating point
generator (a built-in generator). Spirit has many pre-defined generators and
consistent naming conventions will help you finding your way through the maze.
Especially important to note is that things related to identical entities (as
in this case, floating point numbers) are named identically in __karma__ and in
__qi__. Actually, both libraries are using the very same variable instance to
refer to a floating point generator or parser: `double_`.
[heading Trivial Example #2 Generating two numbers]
Now, let's create a generator that will output a line consisting of two
floating-point numbers.
double_ << double_
Here you see the familiar floating-point numeric generator `double_` used twice,
once for each number. If you are used to see the `'>>'` operator for concatenating
two parsers in __qi__ you might wonder, what's that `'<<'` operator doing in
there? We decided to distinguish generating and parsing of sequences the same
way as the std::stream libraries do: we use operator `'>>'` for input (parsing),
and operator `'<<'` for output (generating). Other than that there is no
significant difference. The above program creates a generator from two simpler
generators, glueing them together with the sequence operator. The result is a
generator that is a composition of smaller generators. Whitespace between
numbers can implicitly be inserted depending on how the generator is invoked
(see below).
[note When we combine generators, we end up with a "bigger" generator, but
it's still a generator. Generators can get bigger and bigger, nesting more
and more, but whenever you glue two generators together, you end up with one
bigger generator. This is an important concept.
]
[heading Trivial Example #3 Generating one or more numbers]
Now, creating output for two numbers is not too interesting. Let's create a
generator that will output zero or more floating-point numbers in a row.
*double_
This is like a regular-expression Kleene Star. We moved the `*` to the front for
the same reasons we did in __qi__: we must work with the syntax rules of C++.
But if you know regular expressions (and for sure you remember those C++ syntax
rules) it will start to look very familiar in a matter of a very short time.
Any expression that evaluates to a generator may be used with the Kleene Star.
Keep in mind, though, that due to C++ operator precedence rules you may need
to put the expression in parentheses for complex expressions. As above,
whitespace can be inserted implicitly in between the generated numbers, if
needed.
[heading Trivial Example #4 Generating a comma-delimited list of numbers]
We follow the lead of __qi__'s warming up section and will create a generator
that produces a comma-delimited list of numbers.
double_ << *(lit(',') << double_)
Notice `lit(',')`. It is a literal character generator that simply generates
the comma `','`. In this case, the Kleene Star is modifying a more complex
generator, namely, the one generated by the expression:
(lit(',') << double_)
Note that this is a case where the parentheses are necessary. The Kleene Star
encloses the complete expression above, repeating the whole pattern in the
generated output zero or more times.
[heading Let's Generate!]
We're done with defining the generator. All that's left is to invoke the
generator to do its work. For now, we will use the `generate_delimited` function.
One overload of this function accepts four arguments:
# An output iterator accepting the generated characters
# The generator expression
# Another generator called the delimiting generator
# The data to format and output
While comparing this minimal example with an equivalent parser example we
notice a significant difference. It is possible (and actually, it makes a lot
of sense) to use a parser without creating any internal representation of the
parsed input (i.e. without 'producing' any data from the parsed input). Using
a parser in this mode checks the provided input against
the given parser expression allowing to verify whether the input is parsable.
For generators this mode doesn't make any sense. What is output generation
without generating any output? So we always will have to supply the data the
output should be generated from. In our example we supply a list of `double`
numbers as the last parameter to the function `generate_delimited` (see code
below).
In this example, we wish to delimit the generated numbers by spaces. Another
generator named `space` is included in Spirit's repertoire of predefined
generators. It is a very trivial generator that simply produces spaces. It is
the equivalent to writing `lit(' ')`, or simply `' '`. It has been
implemented for similarity with the corresponding predefined `space` parser.
We will use `space` as our delimiter. The delimiter is the one responsible for
inserting characters in between generator elements such as the `double_` and
`lit`.
Ok, so now let's generate (for the complete source code of this example please
refer to [@../../example/karma/num_list1.cpp num_list1.cpp]).
[import ../../example/karma/num_list1.cpp]
[tutorial_karma_numlist1]
[note You might wonder how a `vector<double>`, which is actually a single data
structure, can be used as an argument (we call it attribute) to a sequence
of generators. This seems to be counter intuitive and doesn't match with
your experience of using `printf`, where each formatting placeholder has
to be matched with a corresponding argument. Well, we will explain this
behavior in more detail later in this tutorial. For now just consider
this to be a special case, implemented on purpose to allow more flexible
output formatting of STL containers: sequences accept a single container
attribute if all elements of this sequence accept attributes compatible
with the elements held by this container.]
The generate function returns `true` or `false` depending on the result of the
output generation. As outlined in different places of this documentation, a
generator may fail for different reasons. One of the possible reasons is an
error in the underlying output iterator (memory exhausted or disk full, etc.).
Another reason might be that the data doesn't match the requirements of a
particular generator.
[note `char` and `wchar_t` operands
The careful reader may notice that the generator expression has `','` instead
of `lit(',')` as the previous examples did. This is ok due to C++ syntax
rules of conversion. Spirit provides `<<` operators that are overloaded to
accept a `char` or `wchar_t` argument on its left or right (but not both).
An operator may be overloaded if at least one of its parameters is a
user-defined type. In this case, the `double_` is the 2nd argument to
`operator<<`, and so the proper overload of `<<` is used, converting `','`
into a character literal generator.
The problem with omitting the `lit` should be obvious: `'a' << 'b'` is not a
spirit generator, it is a numeric expression, left-shifting the ASCII (or
another encoding) value of `'a'` by the ASCII value of `'b'`. However, both
`lit('a') << 'b'` and `'a' << lit('b')` are Spirit sequence generators
for the letter `'a'` followed by `'b'`. You'll get used to it, sooner or
later.
]
Note that we inlined the generator directly in the call to `generate_delimited`.
Upon calling this function, the expression evaluates into a temporary,
unnamed generator which is passed into the `generate_delimited` function,
used, and then destroyed.
Here, we chose to make the generate function generic by making it a template,
parameterized by the output iterator type. By doing so, it can put the generated
data into any STL conforming output iterator.
[endsect] [/ Warming up]

View File

@@ -0,0 +1,52 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lex Lex - Writing Lexical Analyzers]
[include lex/introduction.qbk]
[section:tutorials __lex__ Tutorials]
[include lex/lexer_tutorials.qbk]
[include lex/lexer_quickstart1.qbk]
[include lex/lexer_quickstart2.qbk]
[include lex/lexer_quickstart3.qbk]
[endsect]
[section:abstracts Abstracts]
[section Lexer Primitives]
[/ include lex/lexer_primitives.qbk]
[include lex/tokens_values.qbk]
[/ include lex/token_definition.qbk]
[endsect]
[include lex/tokenizing.qbk]
[include lex/lexer_semantic_actions.qbk]
[include lex/lexer_static_model.qbk]
[/ include lex/parsing_using_a_lexer.qbk]
[/ include lex/lexer_attributes.qbk]
[/ include lex/lexer_states.qbk]
[endsect]
[section:quick_reference Quick Reference]
[include lex/quick_reference.qbk]
[include lex/lexer.qbk]
[endsect]
[section:reference Reference]
[import ../example/lex/reference.cpp]
[include lex/concepts.qbk]
[include lex/basics.qbk]
[include lex/lexer_api.qbk]
[include lex/token_primitives.qbk]
[/ include lex/token.qbk]
[/ include lex/tokendef.qbk]
[/ include lex/lexer_class.qbk]
[/ include lex/token_class.qbk]
[/ include lex/tokendef_class.qbk]
[endsect]
[endsect]

View File

@@ -0,0 +1,40 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lex_basics Basics]
[heading Examples]
All sections in the reference present some real world examples. The
examples use a common test harness to keep the example code as minimal
and direct to the point as possible. The test harness is presented
below.
Some includes:
[reference_lex_includes]
Our test functions:
This one tests token definitions.
[reference_lex_test]
[heading Models]
Predefined models include:
* any literal string, e.g. "Hello, World",
* a pointer/reference to a null-terminated array of characters
* a `std::basic_string<Char>`
The namespace `boost::spirit::traits` is open for users to provide their
own specializations.
[endsect]

View File

@@ -0,0 +1,231 @@
[/==============================================================================
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2001-2011 Joel de Guzman
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_concepts Lexer Concepts]
__lex__ components fall into a couple of generalized __concepts__. The
/Lexer/ is the most fundamental concept. All __lex__ components are
models of the /Lexer/ concept. /PrimitiveLexer/, /UnaryLexer/,
and /NaryLexer/ are all refinements of the /Lexer/ concept.
The following sections provide details on these concepts.
[/////////////////////////////////////////////////////////////////////////////]
[section Lexer]
[heading Description]
The /Lexer/ is the most fundamental concept. A Lexer has a member
function, `collect`, that accepts a token definition container `Def`, and a
the name of the lexer state the token definitions of the lexer component need
to be added to (a string). It doesn't return anything (return type is `void`).
Each Lexer can represent a specific pattern or algorithm, or it
can be a more complex lexer component formed as a composition of other Lexer's.
Additionally, a Lexer exposes a member `add_actions`, that accepts the token
definition container `Def`, while returning nothing (again, the returned type
is `void`).
[variablelist Notation
[[`l`] [A `Lexer`.]]
[[`L`] [A `Lexer` type.]]
[[`Def`] [A token definition container type.]]
[[`State`] [A type used to represent lexer state names.]]
]
[heading Valid Expressions]
In the expressions below, the behavior of the lexer component, `l`, is left
unspecified in the base `Lexer` concept. These are specified in subsequent,
more refined concepts and by the actual models thereof.
For any Lexer the following expressions must be valid:
[table
[[Expression] [Semantics] [Return type]]
[[`l.collect(def, state, targetstate)`]
[Add all token definitions provided
by this Lexer instance to the lexer
state `state` of the token definition
container `def`. After matching this token, the
lexer should be switched into the state
`targetstate` (optional)] [`void`]]
[[`l.add_actions(def)`] [Add all semantic actions provided
by this Lexer instance to the token
definition container `def`.] [`void`]]
]
[heading Type Expressions]
[table
[[Expression] [Description]]
[[`traits::is_lexer<L>::type`] [Metafunction that evaluates to `mpl::true_` if
a certain type, `L` is a Lexer, `mpl::false_`
otherwise (See __mpl_boolean_constant__).]]
]
[heading Postcondition]
Upon return from `l.collect` the following post conditions should hold:
* On return, `def` holds all token definitions defined in the Lexer, `l`. This
includes all Lexer's contained inside `l`.
Upon return from `l.add_actions` the following post conditions should hold:
* On return, `def` holds all semantic actions correctly associated with the
corresponding token definitions as defined in the Lexer, `l`. This
includes all semantic actions defined by the Lexer's contained inside `l`.
[heading Models]
All lexer components in __lex__ are models of the /Lexer/ concept.
[endsect] [/ Lexer Concept]
[/////////////////////////////////////////////////////////////////////////////]
[section PrimitiveLexer]
[heading Description]
/PrimitiveLexer/ is the most basic building block that the client uses
to build more complex lexer components.
[heading Refinement of]
[:__lexer_concept__]
[heading Type Expressions]
[table
[[Expression] [Description]]
[[`traits::is_primitive_lexer<L>::type`] [Metafunction that evaluates to `mpl::true_` if
a certain type, `L`, is a PrimitiveLexer, `mpl::false_`
otherwise (See __mpl_boolean_constant__).]]
]
[heading Models]
The following lexer components conform to this model:
* character literals (i.e. `'x'`), `char_`,
* string literals (`"abc"`), `std::basic_string<>`, `string`
__fixme__ Add more links to /PrimitiveLexer/ models here.
[endsect] [/ PrimitiveLexer Concept]
[/////////////////////////////////////////////////////////////////////////////]
[section UnaryLexer]
[heading Description]
/UnaryLexer/ is a composite lexer component that has a single subject. The
UnaryLexer may change the behavior of its subject following the
__delegate_pattern__.
[heading Refinement of]
[:__lexer_concept__]
[variablelist Notation
[[`l`] [A UnaryLexer.]]
[[`L`] [A UnaryLexer type.]]
]
[heading Valid Expressions]
In addition to the requirements defined in __lexer_concept__, for any
UnaryLexer the following must be met:
[table
[[Expression] [Semantics] [Return type]]
[[`l.subject`] [Subject lexer component.] [__lexer_concept__]]
]
[heading Type Expressions]
[table
[[Expression] [Description]]
[[`L::subject_type`] [The subject lexer component type.]]
[[`traits::is_unary_lexer<L>::type`] [Metafunction that evaluates to `mpl::true_` if
a certain type, `L` is a UnaryLexer, `mpl::false_`
otherwise (See __mpl_boolean_constant__).]]
]
[heading Invariants]
For any UnaryLexer, `L`, the following invariant always holds:
* `traits::is_lexer<L::subject_type>::type` evaluates to `mpl::true_`
[heading Models]
The following lexer components conform to this model:
* action lexer component (allowing to attach semantic actions)
__fixme__ Add more links to models of UnaryLexer concept
[endsect] [/ UnaryLexer Concept]
[/////////////////////////////////////////////////////////////////////////////]
[section NaryLexer]
[heading Description]
/NaryLexer/ is a composite lexer component that has one or more subjects. The
NaryLexer allows its subjects to be treated in the same way as a single
instance of a __lexer_concept__ following the __composite_pattern__.
[heading Refinement of]
[:__lexer_concept__]
[variablelist Notation
[[`l`] [A NaryLexer.]]
[[`L`] [A NaryLexer type.]]
]
[heading Valid Expressions]
In addition to the requirements defined in __lexer_concept__, for any
NaryLexer the following must be met:
[table
[[Expression] [Semantics] [Return type]]
[[`l.elements`] [The tuple of elements.] [A __fusion__ Sequence of __lexer_concept__ types.]]
]
[heading Type Expressions]
[table
[[Expression] [Description]]
[[`l.elements_type`] [Elements tuple type.]]
[[`traits::is_nary_lexer<L>::type`] [Metafunction that evaluates to `mpl::true_` if
a certain type, `L` is a NaryLexer, `mpl::false_`
otherwise (See __mpl_boolean_constant__).]]
]
[heading Invariants]
For each element, `E`, in any NaryLexer, `L`, the following
invariant always holds:
* `traits::is_lexer<E>::type` evaluates to `mpl::true_`
[heading Models]
The following lexer components conform to this model:
* lexer sequence component
__fixme__ Add more links to models of NaryLexer concept
[endsect] [/ NaryLexer Concept]
[endsect]

View File

@@ -0,0 +1,154 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_introduction Introduction to __lex__]
Lexical scanning is the process of analyzing the stream of input characters and
separating it into strings called tokens, separated by whitespace.
Most compiler texts start here, and devote several chapters to discussing
various ways to build scanners. __lex__ is a library built to take care of the
complexities of creating a lexer for your grammar (in this documentation we
will use the terms 'lexical analyzer', 'lexer' and 'scanner' interchangeably).
All that is needed to create a lexer is to know the set of patterns describing
the different tokens you want to recognize in the input. To make this a bit more
formal, here are some definitions:
* A token is a sequence of consecutive characters having a collective meaning.
Tokens may have attributes specific to the token type, carrying additional
information about the matched character sequence.
* A pattern is a rule expressed as a regular expression and describing how a
particular token can be formed. For example, [^\[A-Za-z\]\[A-Za-z_0-9\]*] is
a pattern for a rule matching C++ identifiers.
* Characters between tokens are called whitespace; these include spaces, tabs,
newlines, and formfeeds. Many people also count comments as whitespace,
though since some tools such as lint look at comments, this method is not
perfect.
[heading Why Use a Separate Lexer?]
Typically, lexical scanning is done in a separate module from the parser,
feeding the parser with a stream of input tokens only. Theoretically it is
not necessary implement this separation as in the end there is only one set of
syntactical rules defining the language, so in theory we could write the whole
parser in one module. In fact, __qi__ allows you to write parsers without using a
lexer, parsing the input character stream directly, and for the most part this
is the way __spirit__ has been used since its invention.
However, this separation has both practical and theoretical basis, and proves to
be very useful in practical applications. In 1956, Noam Chomsky defined the
"Chomsky Hierarchy" of grammars:
* Type 0: Unrestricted grammars (e.g., natural languages)
* Type 1: Context-Sensitive grammars
* Type 2: Context-Free grammars
* Type 3: Regular grammars
The complexity of these grammars increases from regular grammars being the
simplest to unrestricted grammars being the most complex. Similarly, the
complexity of pattern recognition for these grammars increases. Although, a few
features of some programming languages (such as C++) are Type 1, fortunately
for the most part programming languages can be described using only the Types 2
and 3. The neat part about these two types is that they are well known and the
ways to parse them are well understood. It has been shown that any regular
grammar can be parsed using a state machine (finite automaton). Similarly,
context-free grammars can always be parsed using a push-down automaton
(essentially a state machine augmented by a stack).
In real programming languages and practical grammars, the parts that can be
handled as regular expressions tend to be the lower-level pieces, such as the
definition of an identifier or of an integer value:
letter := [a-zA-Z]
digit := [0-9]
identifier := letter [ letter | digit ]*
integer := digit+
Higher level parts of practical grammars tend to be more complex and can't be
implemented using plain regular expressions. We need to store
information on the built-in hardware stack while recursing the grammar
hierarchy, and that is the preferred approach used for top-down
parsing. Since it takes a different kind of abstract machine to parse the two
types of grammars, it proved to be efficient to separate the lexical scanner
into a separate module which is built around the idea of a state machine. The
goal here is to use the simplest parsing technique needed for the job.
Another, more practical, reason for separating the scanner from the parser is
the need for backtracking during parsing. The input data is a stream of
characters, which is often thought to be processed left to right without any
backtracking. Unfortunately, in practice most of the time that isn't possible.
Almost every language has certain keywords such as IF, FOR, and WHILE. The
decision if a certain character sequence actually comprises a keyword or just
an identifier often can be made only after seeing the first delimiter /after/
it. In fact, this makes the process backtracking, since we need to store the
string long enough to be able to make the decision. The same is true for more
coarse grained language features such as nested IF/ELSE statements, where the
decision about to which IF belongs the last ELSE statement can be made only
after seeing the whole construct.
So the structure of a conventional compiler often involves splitting up the
functions of the lower-level and higher-level parsing. The lexical scanner
deals with things at the character level, collecting characters into strings,
converting character sequence into different representations as integers, etc.,
and passing them along to the parser proper as indivisible tokens. It's also
considered normal to let the scanner do additional jobs, such as identifying
keywords, storing identifiers in tables, etc.
Now, __spirit__ follows this structure, where __lex__ can be used to implement
state machine based recognizers, while __qi__ can be used to build recognizers
for context free grammars. Since both modules are seamlessly integrated with
each other and with the C++ target language it is even possible to use the
provided functionality to build more complex grammar recognizers.
[heading Advantages of using __lex__]
The advantage of using __lex__ to create the lexical analyzer over using more
traditional tools such as __flex__ is its carefully crafted integration with
the __spirit__ library and the C++ host language. You don't need any external
tools to generate the code, your lexer will be perfectly integrated with the
rest of your program, making it possible to freely access any context
information and data structure. Since the C++ compiler sees all the code, it
will generate optimal code no matter what configuration options have been chosen
by the user. __lex__ gives you the vast majority of features you could get from
a similar __flex__ program without the need to leave C++ as a host language:
* The definition of tokens is done using regular expressions (patterns)
* The token definitions can refer to special substitution strings (pattern
macros) simplifying pattern definitions
* The generated lexical scanner may have multiple start states
* It is possible to attach code to any of the token definitions; this code gets
executed whenever the corresponding token pattern has been matched
Even if it is possible to use __lex__ to generate C++ code representing
the lexical analyzer (we will refer to that as the /static/ model, described in
more detail in the section __sec_lex_static_model__) - a model
very similar to the way __flex__ operates - we will mainly focus on the
opposite, the /dynamic/ model. You can directly integrate the token definitions
into your C++ program, building the lexical analyzer dynamically at runtime. The
dynamic model is something not supported by __flex__ or other lexical scanner
generators (such as __re2c__, __ragel__, etc.). This dynamic flexibility allows
you to speed up the development of your application.
[heading The Library Structure of __lex__]
The [link spirit.lexerflow figure] below shows a high level overview of how the
__lex__ library might be used in an application. __lex__ allows to create
lexical analyzers based on patterns. These patterns are regular expression
based rules used to define the different tokens to be recognized in the
character input sequence. The input sequence is expected to be provided to the
lexical analyzer as an arbitrary standard forward iterator. The lexical
analyzer itself exposes a standard forward iterator as well. The difference
here is that the exposed iterator provides access to the token sequence instead
of to the character sequence. The tokens in this sequence are constructed on
the fly by analyzing the underlying character sequence and matching this to the
patterns as defined by the application.
[fig lexerflow.png..The Library structure and Common Flow of Information while using __lex__ in an application..spirit.lexerflow]
[endsect]

View File

@@ -0,0 +1,104 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer Supported Regular Expressions]
[table Regular expressions support
[[Expression] [Meaning]]
[[`x`] [Match any character `x`]]
[[`.`] [Match any except newline (or optionally *any* character)]]
[[`"..."`] [All characters taken as literals between double quotes, except escape sequences]]
[[`[xyz]`] [A character class; in this case matches `x`, `y` or `z`]]
[[`[abj-oZ]`] [A character class with a range in it; matches `a`, `b` any
letter from `j` through `o` or a `Z`]]
[[`[^A-Z]`] [A negated character class i.e. any character but those in
the class. In this case, any character except an uppercase
letter]]
[[`r*`] [Zero or more r's (greedy), where r is any regular expression]]
[[`r*?`] [Zero or more r's (abstemious), where r is any regular expression]]
[[`r+`] [One or more r's (greedy)]]
[[`r+?`] [One or more r's (abstemious)]]
[[`r?`] [Zero or one r's (greedy), i.e. optional]]
[[`r??`] [Zero or one r's (abstemious), i.e. optional]]
[[`r{2,5}`] [Anywhere between two and five r's (greedy)]]
[[`r{2,5}?`] [Anywhere between two and five r's (abstemious)]]
[[`r{2,}`] [Two or more r's (greedy)]]
[[`r{2,}?`] [Two or more r's (abstemious)]]
[[`r{4}`] [Exactly four r's]]
[[`{NAME}`] [The macro `NAME` (see below)]]
[[`"[xyz]\"foo"`] [The literal string `[xyz]\"foo`]]
[[`\X`] [If X is `a`, `b`, `e`, `n`, `r`, `f`, `t`, `v` then the
ANSI-C interpretation of `\x`. Otherwise a literal `X`
(used to escape operators such as `*`)]]
[[`\0`] [A NUL character (ASCII code 0)]]
[[`\123`] [The character with octal value 123]]
[[`\x2a`] [The character with hexadecimal value 2a]]
[[`\cX`] [A named control character `X`.]]
[[`\a`] [A shortcut for Alert (bell).]]
[[`\b`] [A shortcut for Backspace]]
[[`\e`] [A shortcut for ESC (escape character `0x1b`)]]
[[`\n`] [A shortcut for newline]]
[[`\r`] [A shortcut for carriage return]]
[[`\f`] [A shortcut for form feed `0x0c`]]
[[`\t`] [A shortcut for horizontal tab `0x09`]]
[[`\v`] [A shortcut for vertical tab `0x0b`]]
[[`\d`] [A shortcut for `[0-9]`]]
[[`\D`] [A shortcut for `[^0-9]`]]
[[`\s`] [A shortcut for `[\x20\t\n\r\f\v]`]]
[[`\S`] [A shortcut for `[^\x20\t\n\r\f\v]`]]
[[`\w`] [A shortcut for `[a-zA-Z0-9_]`]]
[[`\W`] [A shortcut for `[^a-zA-Z0-9_]`]]
[[`(r)`] [Match an `r`; parenthesis are used to override precedence
(see below)]]
[[`(?r-s:pattern)`] [apply option 'r' and omit option 's' while interpreting pattern.
Options may be zero or more of the characters 'i' or 's'.
'i' means case-insensitive. '-i' means case-sensitive.
's' alters the meaning of the '.' syntax to match any single character whatsoever.
'-s' alters the meaning of '.' to match any character except '`\n`'.]]
[[`rs`] [The regular expression `r` followed by the regular
expression `s` (a sequence)]]
[[`r|s`] [Either an `r` or and `s`]]
[[`^r`] [An `r` but only at the beginning of a line (i.e. when just
starting to scan, or right after a newline has been
scanned)]]
[[`r`$] [An `r` but only at the end of a line (i.e. just before a
newline)]]
]
[note POSIX character classes are not currently supported, due to performance issues
when creating them in wide character mode.]
[tip If you want to build tokens for syntaxes that recognize items like quotes
(`"'"`, `'"'`) and backslash (`\`), here is example syntax to get you started.
The lesson here really is to remember that both c++, as well as regular
expressions require escaping with `\` for some constructs, which can
cascade.
``
quote1 = "'"; // match single "'"
quote2 = "\\\""; // match single '"'
literal_quote1 = "\\'"; // match backslash followed by single "'"
literal_quote2 = "\\\\\\\""; // match backslash followed by single '"'
literal_backslash = "\\\\\\\\"; // match two backslashes
``
]
[heading Regular Expression Precedence]
* `rs` has highest precedence
* `r*` has next highest (`+`, `?`, `{n,m}` have the same precedence as `*`)
* `r|s` has the lowest precedence
[heading Macros]
Regular expressions can be given a name and referred to in rules using the
syntax `{NAME}` where `NAME` is the name you have given to the macro. A macro
name can be at most 30 characters long and must start with a `_` or a letter.
Subsequent characters can be `_`, `-`, a letter or a decimal digit.
[endsect]

View File

@@ -0,0 +1,189 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_api Lexer API]
[heading Description]
The library provides a couple of free functions to make using the lexer a snap.
These functions have three forms. The first form, `tokenize`, simplifies the
usage of a stand alone lexer (without parsing). The second form,
`tokenize_and_parse`, combines a lexer step with parsing on
the token level (without a skipper). The third, `tokenize_and_phrase_parse`,
works on the token level as well, but additionally employs a skip parser. The
latter two versions can take in attributes by reference that will hold the
parsed values on a successful parse.
[heading Header]
// forwards to <boost/spirit/home/lex/tokenize_and_parse.hpp>
#include <boost/spirit/include/lex_tokenize_and_parse.hpp>
For variadic attributes:
// forwards to <boost/spirit/home/lex/tokenize_and_parse_attr.hpp>
#include <boost/spirit/include/lex_tokenize_and_parse_attr.hpp>
The variadic attributes version of the API allows one or more
attributes to be passed into the API functions. The functions taking two
or more attributes are usable when the parser expression is a
__qi_sequence__ only. In this case each of the
attributes passed have to match the corresponding part of the sequence.
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`boost::spirit::lex::tokenize` ]]
[[`boost::spirit::lex::tokenize_and_parse` ]]
[[`boost::spirit::lex::tokenize_and_phrase_parse` ]]
[[`boost::spirit::qi::skip_flag::postskip` ]]
[[`boost::spirit::qi::skip_flag::dont_postskip` ]]
]
[heading Synopsis]
The `tokenize` function is one of the main lexer API functions. It
simplifies using a lexer to tokenize a given input sequence. It's main
purpose is to use the lexer to tokenize all the input.
Both functions take a pair of iterators spanning the underlying input
stream to scan, the lexer object (built from the token definitions),
and an (optional) functor being called for each of the generated tokens. If no
function object `f` is given, the generated tokens will be discarded.
The functions return `true` if the scanning of the input succeeded (the
given input sequence has been successfully matched by the given
token definitions).
The argument `f` is expected to be a function (callable) object taking a single
argument of the token type and returning a bool, indicating whether
the tokenization should be canceled. If it returns `false` the function
`tokenize` will return `false` as well.
The `initial_state` argument forces lexing to start with the given lexer state.
If this is omitted lexing starts in the `"INITIAL"` state.
template <typename Iterator, typename Lexer>
inline bool
tokenize(
Iterator& first
, Iterator last
, Lexer const& lex
, typename Lexer::char_type const* initial_state = 0);
template <typename Iterator, typename Lexer, typename F>
inline bool
tokenize(
Iterator& first
, Iterator last
, Lexer const& lex
, F f
, typename Lexer::char_type const* initial_state = 0);
The `tokenize_and_parse` function is one of the main lexer API
functions. It simplifies using a lexer as the underlying token source
while parsing a given input sequence.
The functions take a pair of iterators spanning the underlying input
stream to parse, the lexer object (built from the token definitions)
and a parser object (built from the parser grammar definition). Additionally
they may take the attributes for the parser step.
The function returns `true` if the parsing succeeded (the given input
sequence has been successfully matched by the given grammar).
template <typename Iterator, typename Lexer, typename ParserExpr>
inline bool
tokenize_and_parse(
Iterator& first
, Iterator last
, Lexer const& lex
, ParserExpr const& expr)
template <typename Iterator, typename Lexer, typename ParserExpr
, typename Attr1, typename Attr2, ..., typename AttrN>
inline bool
tokenize_and_parse(
Iterator& first
, Iterator last
, Lexer const& lex
, ParserExpr const& expr
, Attr1 const& attr1, Attr2 const& attr2, ..., AttrN const& attrN);
The functions `tokenize_and_phrase_parse` take a pair of iterators spanning
the underlying input stream to parse, the lexer object (built from the token
definitions) and a parser object (built from the parser grammar definition).
The additional skipper parameter will be used as the skip parser during
the parsing process. Additionally they may take the attributes for the parser
step.
The function returns `true` if the parsing succeeded (the given input
sequence has been successfully matched by the given grammar).
template <typename Iterator, typename Lexer, typename ParserExpr
, typename Skipper>
inline bool
tokenize_and_phrase_parse(
Iterator& first
, Iterator last
, Lexer const& lex
, ParserExpr const& expr
, Skipper const& skipper
, BOOST_SCOPED_ENUM(skip_flag) post_skip = skip_flag::postskip);
template <typename Iterator, typename Lexer, typename ParserExpr
, typename Skipper, typename Attribute>
inline bool
tokenize_and_phrase_parse(
Iterator& first
, Iterator last
, Lexer const& lex
, ParserExpr const& expr
, Skipper const& skipper
, Attribute& attr);
template <typename Iterator, typename Lexer, typename ParserExpr
, typename Skipper, typename Attribute>
inline bool
tokenize_and_phrase_parse(
Iterator& first
, Iterator last
, Lexer const& lex
, ParserExpr const& expr
, Skipper const& skipper
, BOOST_SCOPED_ENUM(skip_flag) post_skip, Attribute& attr);
The maximum number of supported arguments is limited by the preprocessor
constant `SPIRIT_ARGUMENTS_LIMIT`. This constant defaults to the value defined
by the preprocessor constant `PHOENIX_LIMIT` (which in turn defaults to `10`).
[note The variadic function with two or more attributes internally combine
references to all passed attributes into a `fusion::vector` and forward
this as a combined attribute to the corresponding one attribute function.]
The `tokenize_and_phrase_parse` functions not taking an explicit `skip_flag`
as one of their arguments invoke the passed skipper after a successful match
of the parser expression. This can be inhibited by using the other versions of
that function while passing `skip_flag::dont_postskip` to the corresponding
argument.
[heading Template parameters]
[table
[[Parameter] [Description]]
[[`Iterator`] [__fwditer__ pointing to the underlying input sequence to parse.]]
[[`Lexer`] [A lexer (token definition) object.]]
[[`F`] [A function object called for each generated token.]]
[[`ParserExpr`] [An expression that can be converted to a Qi parser.]]
[[`Skipper`] [Parser used to skip white spaces.]]
[[`Attr1`, `Attr2`, ..., `AttrN`][One or more attributes.]]
]
[endsect]

View File

@@ -0,0 +1,12 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_attributes Lexer Attributes]
[endsect]

View File

@@ -0,0 +1,19 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer Lexer Class]
[heading The lexertl::lexer Class Implementing the Dynamic Model]
[heading The lexertl::actor_lexer Class Implementing the Dynamic Model]
[heading The lexertl::static_lexer Class Implementing the Static Model]
[heading The lexertl::static_actor_lexer Class Implementing the Static Model]
[endsect]

View File

@@ -0,0 +1,16 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_primitives Lexer Primitives]
[/ Describe the primitive lexer constructs, such as token_def, lexer ]
[/ Describe the primitive lexer constructs usable in parsers, such as
in_state[], set_state(), token(), tokenid(), etc. ]
[endsect]

View File

@@ -0,0 +1,97 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_quickstart1 Quickstart 1 - A word counter using __lex__]
__lex__ is very modular, which follows the general building principle of the
__spirit__ libraries. You never pay for features you don't use. It is nicely
integrated with the other parts of __spirit__ but nevertheless can be used
separately to build stand alone lexical analyzers.
The first quick start example describes a stand alone application:
counting characters, words, and lines in a file, very similar to what the well
known Unix command `wc` is doing (for the full example code see here:
[@../../example/lex/word_count_functor.cpp word_count_functor.cpp]).
[import ../example/lex/word_count_functor.cpp]
[heading Prerequisites]
The only required `#include` specific to /Spirit.Lex/ follows. It is a wrapper
for all necessary definitions to use /Spirit.Lex/ in a stand alone fashion, and
on top of the __lexertl__ library. Additionally we `#include` two of the Boost
headers to define `boost::bind()` and `boost::ref()`.
[wcf_includes]
To make all the code below more readable we introduce the following namespaces.
[wcf_namespaces]
[heading Defining Tokens]
The most important step while creating a lexer using __lex__ is to define the
tokens to be recognized in the input sequence. This is normally done by
defining the regular expressions describing the matching character sequences,
and optionally their corresponding token ids. Additionally the defined tokens
need to be associated with an instance of a lexer object as provided by the
library. The following code snippet shows how this can be done using __lex__.
[wcf_token_definition]
[heading Doing the Useful Work]
We will use a setup, where we want the __lex__ library to invoke a given
function after any of the generated tokens is recognized. For this reason
we need to implement a functor taking at least the generated token as an
argument and returning a boolean value allowing to stop the tokenization
process. The default token type used in this example carries a token value of
the type __boost_iterator_range__`<BaseIterator>` pointing to the matched
range in the underlying input sequence.
[wcf_functor]
All what is left is to write some boilerplate code helping to tie together the
pieces described so far. To simplify this example we call the `lex::tokenize()`
function implemented in __lex__ (for a more detailed description of this
function see here: __fixme__), even if we could have written a loop to iterate
over the lexer iterators [`first`, `last`) as well.
[heading Pulling Everything Together]
[wcf_main]
[heading Comparing __lex__ with __flex__]
This example was deliberately chosen to be as much as possible similar to the
equivalent __flex__ program (see below), which isn't too different from what
has to be written when using __lex__.
[note Interestingly enough, performance comparisons of lexical analyzers
written using __lex__ with equivalent programs generated by
__flex__ show that both have comparable execution speeds!
Generally, thanks to the highly optimized __lexertl__ library and
due its carefully designed integration with __spirit__ the
abstraction penalty to be paid for using __lex__ is negligible.
]
The remaining examples in this tutorial will use more sophisticated features
of __lex__, mainly to allow further simplification of the code to be written,
while maintaining the similarity with corresponding features of __flex__.
__lex__ has been designed to be as similar to __flex__ as possible. That
is why this documentation will provide the corresponding __flex__ code for the
shown __lex__ examples almost everywhere. So consequently, here is the __flex__
code corresponding to the example as shown above.
[wcf_flex_version]
[endsect]

View File

@@ -0,0 +1,103 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_quickstart2 Quickstart 2 - A better word counter using __lex__]
People familiar with __flex__ will probably complain about the example from the
section __sec_lex_quickstart_1__ as being overly complex and not being
written to leverage the possibilities provided by this tool. In particular the
previous example did not directly use the lexer actions to count the lines,
words, and characters. So the example provided in this step of the tutorial will
show how to use semantic actions in __lex__. Even though this examples still
counts textual elements, the purpose is to introduce new concepts and
configuration options along the lines (for the full example code
see here: [@../../example/lex/word_count_lexer.cpp word_count_lexer.cpp]).
[import ../example/lex/word_count_lexer.cpp]
[heading Prerequisites]
In addition to the only required `#include` specific to /Spirit.Lex/ this
example needs to include a couple of header files from the __phoenix__
library. This example shows how to attach functors to token definitions, which
could be done using any type of C++ technique resulting in a callable object.
Using __phoenix__ for this task simplifies things and avoids adding
dependencies to other libraries (__phoenix__ is already in use for
__spirit__ anyway).
[wcl_includes]
To make all the code below more readable we introduce the following namespaces.
[wcl_namespaces]
To give a preview at what to expect from this example, here is the flex program
which has been used as the starting point. The useful code is directly included
inside the actions associated with each of the token definitions.
[wcl_flex_version]
[heading Semantic Actions in __lex__]
__lex__ uses a very similar way of associating actions with the token
definitions (which should look familiar to anybody knowledgeable with
__spirit__ as well): specifying the operations to execute inside of a pair of
`[]` brackets. In order to be able to attach semantic actions to token
definitions for each of them there is defined an instance of a `token_def<>`.
[wcl_token_definition]
The semantics of the shown code is as follows. The code inside the `[]`
brackets will be executed whenever the corresponding token has been matched by
the lexical analyzer. This is very similar to __flex__, where the action code
associated with a token definition gets executed after the recognition of a
matching input sequence. The code above uses function objects constructed using
__phoenix__, but it is possible to insert any C++ function or function object
as long as it exposes the proper interface. For more details on please refer
to the section __sec_lex_semactions__.
[heading Associating Token Definitions with the Lexer]
If you compare this code to the code from __sec_lex_quickstart_1__ with regard
to the way how token definitions are associated with the lexer, you will notice
a different syntax being used here. In the previous example we have been
using the `self.add()` style of the API, while we here directly assign the token
definitions to `self`, combining the different token definitions using the `|`
operator. Here is the code snippet again:
this->self
= word [++ref(w), ref(c) += distance(_1)]
| eol [++ref(c), ++ref(l)]
| any [++ref(c)]
;
This way we have a very powerful and natural way of building the lexical
analyzer. If translated into English this may be read as: The lexical analyzer
will recognize ('`=`') tokens as defined by any of ('`|`') the token
definitions `word`, `eol`, and `any`.
A second difference to the previous example is that we do not explicitly
specify any token ids to use for the separate tokens. Using semantic actions to
trigger some useful work has freed us from the need to define those. To ensure
every token gets assigned a id the __lex__ library internally assigns unique
numbers to the token definitions, starting with the constant defined by
`boost::spirit::lex::min_token_id`.
[heading Pulling everything together]
In order to execute the code defined above we still need to instantiate an
instance of the lexer type, feed it from some input sequence and create a pair
of iterators allowing to iterate over the token sequence as created by the
lexer. This code shows how to achieve these steps:
[wcl_main]
[endsect]

View File

@@ -0,0 +1,141 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_quickstart3 Quickstart 3 - Counting Words Using a Parser]
The whole purpose of integrating __lex__ as part of the __spirit__ library was
to add a library allowing the merger of lexical analysis with the parsing
process as defined by a __spirit__ grammar. __spirit__ parsers read their input
from an input sequence accessed by iterators. So naturally, we chose iterators
to be used as the interface between the lexer and the parser. A second goal of
the lexer/parser integration was to enable the usage of different
lexical analyzer libraries. The utilization of iterators seemed to be the
right choice from this standpoint as well, mainly because these can be used as
an abstraction layer hiding implementation specifics of the used lexer
library. The [link spirit.lex.flowcontrol picture] below shows the common
flow control implemented while parsing combined with lexical analysis.
[fig flowofcontrol.png..The common flow control implemented while parsing combined with lexical analysis..spirit.lex.flowcontrol]
Another problem related to the integration of the lexical analyzer with the
parser was to find a way how the defined tokens syntactically could be blended
with the grammar definition syntax of __spirit__. For tokens defined as
instances of the `token_def<>` class the most natural way of integration was
to allow to directly use these as parser components. Semantically these parser
components succeed matching their input whenever the corresponding token type
has been matched by the lexer. This quick start example will demonstrate this
(and more) by counting words again, simply by adding up the numbers inside
of semantic actions of a parser (for the full example code see here:
[@../../example/lex/word_count.cpp word_count.cpp]).
[import ../example/lex/word_count.cpp]
[heading Prerequisites]
This example uses two of the __spirit__ library components: __lex__ and __qi__,
consequently we have to `#include` the corresponding header files. Again, we
need to include a couple of header files from the __phoenix__ library. This
example shows how to attach functors to parser components, which
could be done using any type of C++ technique resulting in a callable object.
Using __phoenix__ for this task simplifies things and avoids adding
dependencies to other libraries (__phoenix__ is already in use for
__spirit__ anyway).
[wcp_includes]
To make all the code below more readable we introduce the following namespaces.
[wcp_namespaces]
[heading Defining Tokens]
If compared to the two previous quick start examples (__sec_lex_quickstart_1__
and __sec_lex_quickstart_2__) the token definition class for this example does
not reveal any surprises. However, it uses lexer token definition macros to
simplify the composition of the regular expressions, which will be described in
more detail in the section __fixme__. Generally, any token definition is usable
without modification from either a stand alone lexical analyzer or in conjunction
with a parser.
[wcp_token_definition]
[heading Using Token Definition Instances as Parsers]
While the integration of lexer and parser in the control flow is achieved by
using special iterators wrapping the lexical analyzer, we still need a means of
expressing in the grammar what tokens to match and where. The token definition
class above uses three different ways of defining a token:
* Using an instance of a `token_def<>`, which is handy whenever you need to
specify a token attribute (for more information about lexer related
attributes please look here: __sec_lex_attributes__).
* Using a single character as the token, in this case the character represents
itself as a token, where the token id is the ASCII character value.
* Using a regular expression represented as a string, where the token id needs
to be specified explicitly to make the token accessible from the grammar
level.
All three token definition methods require a different method of grammar
integration. But as you can see from the following code snippet, each of these
methods are straightforward and blend the corresponding token instances
naturally with the surrounding __qi__ grammar syntax.
[table
[[Token definition] [Parser integration]]
[[`token_def<>`] [The `token_def<>` instance is directly usable as a
parser component. Parsing of this component will
succeed if the regular expression used to define
this has been matched successfully.]]
[[single character] [The single character is directly usable in the
grammar. However, under certain circumstances it needs
to be wrapped by a `char_()` parser component.
Parsing of this component will succeed if the
single character has been matched.]]
[[explicit token id] [To use an explicit token id in a __qi__ grammar you
are required to wrap it with the special `token()`
parser component. Parsing of this component will
succeed if the current token has the same token
id as specified in the expression `token(<id>)`.]]
]
The grammar definition below uses each of the three types demonstrating their
usage.
[wcp_grammar_definition]
As already described (see: __sec_attributes__), the __qi__ parser
library builds upon a set of fully attributed parser components.
Consequently, all token definitions support this attribute model as well. The
most natural way of implementing this was to use the token values as
the attributes exposed by the parser component corresponding to the token
definition (you can read more about this topic here: __sec_lex_tokenvalues__).
The example above takes advantage of the full integration of the token values
as the `token_def<>`'s parser attributes: the `word` token definition is
declared as a `token_def<std::string>`, making every instance of a `word` token
carry the string representation of the matched input sequence as its value.
The semantic action attached to `tok.word` receives this string (represented by
the `_1` placeholder) and uses it to calculate the number of matched
characters: `ref(c) += size(_1)`.
[heading Pulling Everything Together]
The main function needs to implement a bit more logic now as we have to
initialize and start not only the lexical analysis but the parsing process as
well. The three type definitions (`typedef` statements) simplify the creation
of the lexical analyzer and the grammar. After reading the contents of the
given file into memory it calls the function __api_tokenize_and_parse__ to
initialize the lexical analysis and parsing processes.
[wcp_main]
[endsect]

View File

@@ -0,0 +1,186 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_semantic_actions Lexer Semantic Actions]
The main task of a lexer normally is to recognize tokens in the input.
Traditionally this has been complemented with the possibility to execute
arbitrary code whenever a certain token has been detected. __lex__ has been
designed to support this mode of operation as well. We borrow from the concept
of semantic actions for parsers (__qi__) and generators (__karma__). Lexer
semantic actions may be attached to any token definition. These are C++
functions or function objects that are called whenever a token definition
successfully recognizes a portion of the input. Say you have a token definition
`D`, and a C++ function `f`, you can make the lexer call `f` whenever it matches
an input by attaching `f`:
D[f]
The expression above links `f` to the token definition, `D`. The required
prototype of `f` is:
void f (Iterator& start, Iterator& end, pass_flag& matched, Idtype& id, Context& ctx);
[variablelist where:
[[`Iterator& start`] [This is the iterator pointing to the begin of the
matched range in the underlying input sequence. The
type of the iterator is the same as specified while
defining the type of the `lexertl::actor_lexer<...>`
(its first template parameter). The semantic action
is allowed to change the value of this iterator
influencing, the matched input sequence.]]
[[`Iterator& end`] [This is the iterator pointing to the end of the
matched range in the underlying input sequence. The
type of the iterator is the same as specified while
defining the type of the `lexertl::actor_lexer<...>`
(its first template parameter). The semantic action
is allowed to change the value of this iterator
influencing, the matched input sequence.]]
[[`pass_flag& matched`] [This value is pre/initialized to `pass_normal`.
If the semantic action sets it to `pass_fail` this
behaves as if the token has not been matched in
the first place. If the semantic action sets this
to `pass_ignore` the lexer ignores the current
token and tries to match a next token from the
input.]]
[[`Idtype& id`] [This is the token id of the type Idtype (most of
the time this will be a `std::size_t`) for the
matched token. The semantic action is allowed to
change the value of this token id, influencing the
if of the created token.]]
[[`Context& ctx`] [This is a reference to a lexer specific,
unspecified type, providing the context for the
current lexer state. It can be used to access
different internal data items and is needed for
lexer state control from inside a semantic
action.]]
]
When using a C++ function as the semantic action the following prototypes are
allowed as well:
void f (Iterator& start, Iterator& end, pass_flag& matched, Idtype& id);
void f (Iterator& start, Iterator& end, pass_flag& matched);
void f (Iterator& start, Iterator& end);
void f ();
[important In order to use lexer semantic actions you need to use type
`lexertl::actor_lexer<>` as your lexer class (instead of the
type `lexertl::lexer<>` as described in earlier examples).]
[heading The context of a lexer semantic action]
The last parameter passed to any lexer semantic action is a reference to an
unspecified type (see the `Context` type in the table above). This type is
unspecified because it depends on the token type returned by the lexer. It is
implemented in the internals of the iterator type exposed by the lexer.
Nevertheless, any context type is expected to expose a couple of
functions allowing to influence the behavior of the lexer. The following table
gives an overview and a short description of the available functionality.
[table Functions exposed by any context passed to a lexer semantic action
[[Name] [Description]]
[[`Iterator const& get_eoi() const`]
[The function `get_eoi()` may be used by to access the end iterator of
the input stream the lexer has been initialized with]]
[[`void more()`]
[The function `more()` tells the lexer that the next time it matches a
rule, the corresponding token should be appended onto the current token
value rather than replacing it.]]
[[`Iterator const& less(Iterator const& it, int n)`]
[The function `less()` returns an iterator positioned to the nth input
character beyond the current token start iterator (i.e. by passing the
return value to the parameter `end` it is possible to return all but the
first n characters of the current token back to the input stream.]]
[[`bool lookahead(std::size_t id)`]
[The function `lookahead()` can be used to implement lookahead for lexer
engines not supporting constructs like flex' `a/b`
(match `a`, but only when followed by `b`). It invokes the lexer on the
input following the current token without actually moving forward in the
input stream. The function returns whether the lexer was able to match a
token with the given token-id `id`.]]
[[`std::size_t get_state() const` and `void set_state(std::size_t state)`]
[The functions `get_state()` and `set_state()` may be used to introspect
and change the current lexer state.]]
[[`token_value_type get_value() const` and `void set_value(Value const&)`]
[The functions `get_value()` and `set_value()` may be used to introspect
and change the current token value.]]
]
[heading Lexer Semantic Actions Using Phoenix]
Even if it is possible to write your own function object implementations (i.e.
using Boost.Lambda or Boost.Bind), the preferred way of defining lexer semantic
actions is to use __phoenix__. In this case you can access the parameters
described above by using the predefined __spirit__ placeholders:
[table Predefined Phoenix placeholders for lexer semantic actions
[[Placeholder] [Description]]
[[`_start`]
[Refers to the iterator pointing to the beginning of the matched input
sequence. Any modifications to this iterator value will be reflected in
the generated token.]]
[[`_end`]
[Refers to the iterator pointing past the end of the matched input
sequence. Any modifications to this iterator value will be reflected in
the generated token.]]
[[`_pass`]
[References the value signaling the outcome of the semantic action. This
is pre-initialized to `lex::pass_flags::pass_normal`. If this is set to
`lex::pass_flags::pass_fail`, the lexer will behave as if no token has
been matched, if is set to `lex::pass_flags::pass_ignore`, the lexer will
ignore the current match and proceed trying to match tokens from the
input.]]
[[`_tokenid`]
[Refers to the token id of the token to be generated. Any modifications
to this value will be reflected in the generated token.]]
[[`_val`]
[Refers to the value the next token will be initialized from. Any
modifications to this value will be reflected in the generated token.]]
[[`_state`]
[Refers to the lexer state the input has been match in. Any modifications
to this value will be reflected in the lexer itself (the next match will
start in the new state). The currently generated token is not affected
by changes to this variable.]]
[[`_eoi`]
[References the end iterator of the overall lexer input. This value
cannot be changed.]]
]
The context object passed as the last parameter to any lexer semantic action is
not directly accessible while using __phoenix__ expressions. We rather provide
predefined Phoenix functions allowing to invoke the different support functions
as mentioned above. The following table lists the available support functions
and describes their functionality:
[table Support functions usable from Phoenix expressions inside lexer semantic actions
[[Plain function] [Phoenix function] [Description]]
[[`ctx.more()`]
[`more()`]
[The function `more()` tells the lexer that the next time it matches a
rule, the corresponding token should be appended onto the current token
value rather than replacing it.]]
[[`ctx.less()`]
[`less(n)`]
[The function `less()` takes a single integer parameter `n` and returns an
iterator positioned to the nth input character beyond the current token
start iterator (i.e. by assigning the return value to the placeholder
`_end` it is possible to return all but the first `n` characters of the
current token back to the input stream.]]
[[`ctx.lookahead()`]
[`lookahead(std::size_t)` or `lookahead(token_def)`]
[The function `lookahead()` takes a single parameter specifying the token
to match in the input. The function can be used for instance to implement
lookahead for lexer engines not supporting constructs like flex' `a/b`
(match `a`, but only when followed by `b`). It invokes the lexer on the
input following the current token without actually moving forward in the
input stream. The function returns whether the lexer was able to match
the specified token.]]
]
[endsect]

View File

@@ -0,0 +1,21 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_states Lexer States]
[heading Controlling the Lexer State from Lexer Semantic Actions]
[heading Controlling the Lexer State from Parser Semantic Actions]
[heading Using a Lexer State for the Skip Parser]
[endsect]

View File

@@ -0,0 +1,137 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_static_model The /Static/ Lexer Model]
The documentation of __lex__ so far mostly was about describing the features of
the /dynamic/ model, where the tables needed for lexical analysis are generated
from the regular expressions at runtime. The big advantage of the dynamic model
is its flexibility, and its integration with the __spirit__ library and the C++
host language. Its big disadvantage is the need to spend additional runtime to
generate the tables, which especially might be a limitation for larger lexical
analyzers. The /static/ model strives to build upon the smooth integration with
__spirit__ and C++, and reuses large parts of the __lex__ library as described
so far, while overcoming the additional runtime requirements by using
pre-generated tables and tokenizer routines. To make the code generation as
simple as possible, the static model reuses the token definition types developed
for the /dynamic/ model without any changes. As will be shown in this
section, building a code generator based on an existing token definition type
is a matter of writing 3 lines of code.
Assuming you already built a dynamic lexer for your problem, there are two more
steps needed to create a static lexical analyzer using __lex__:
# generating the C++ code for the static analyzer (including the tokenization
function and corresponding tables), and
# modifying the dynamic lexical analyzer to use the generated code.
Both steps are described in more detail in the two sections below (for the full
source code used in this example see the code here:
[@../../example/lex/static_lexer/word_count_tokens.hpp the common token definition],
[@../../example/lex/static_lexer/word_count_generate.cpp the code generator],
[@../../example/lex/static_lexer/word_count_static.hpp the generated code], and
[@../../example/lex/static_lexer/word_count_static.cpp the static lexical analyzer]).
[import ../example/lex/static_lexer/word_count_tokens.hpp]
[import ../example/lex/static_lexer/word_count_static.cpp]
[import ../example/lex/static_lexer/word_count_generate.cpp]
But first we provide the code snippets needed to further understand the
descriptions. Both, the definition of the used token identifier and the of the
token definition class in this example are put into a separate header file to
make these available to the code generator and the static lexical analyzer.
[wc_static_tokenids]
The important point here is, that the token definition class is not different
from a similar class to be used for a dynamic lexical analyzer. The library
has been designed in a way, that all components (dynamic lexical analyzer, code
generator, and static lexical analyzer) can reuse the very same token definition
syntax.
[wc_static_tokendef]
The only thing changing between the three different use cases is the template
parameter used to instantiate a concrete token definition. For the dynamic
model and the code generator you probably will use the __class_lexertl_lexer__
template, where for the static model you will use the
__class_lexertl_static_lexer__ type as the template parameter.
This example not only shows how to build a static lexer, but it additionally
demonstrates how such a lexer can be used for parsing in conjunction with a
__qi__ grammar. For completeness, we provide the simple grammar used in this
example. As you can see, this grammar does not have any dependencies on the
static lexical analyzer, and for this reason it is not different from a grammar
used either without a lexer or using a dynamic lexical analyzer as described
before.
[wc_static_grammar]
[heading Generating the Static Analyzer]
The first additional step to perform in order to create a static lexical
analyzer is to create a small stand alone program for creating the lexer tables
and the corresponding tokenization function. For this purpose the __lex__
library exposes a special API - the function __api_generate_static__. It
implements the whole code generator, no further code is needed. All what it
takes to invoke this function is to supply a token definition instance, an
output stream to use to generate the code to, and an optional string to be used
as a suffix for the name of the generated function. All in all just a couple
lines of code.
[wc_static_generate_main]
The shown code generator will generate output, which should be stored in a file
for later inclusion into the static lexical analyzer as shown in the next
topic (the full generated code can be viewed
[@../../example/lex/static_lexer/word_count_static.hpp here]).
[note The generated code will have compiled in the version number of the
current __lex__ library. This version number is used at compilation time
of your static lexer object to ensure this is compiled using exactly the
same version of the __lex__ library as the lexer tables have been
generated with. If the versions do not match you will see an compilation
error mentioning an `incompatible_static_lexer_version`.
]
[heading Modifying the Dynamic Analyzer]
The second required step to convert an existing dynamic lexer into a static one
is to change your main program at two places. First, you need to change the
type of the used lexer (that is the template parameter used while instantiating
your token definition class). While in the dynamic model we have been using the
__class_lexertl_lexer__ template, we now need to change that to the
__class_lexertl_static_lexer__ type. The second change is tightly related to
the first one and involves correcting the corresponding `#include` statement to:
[wc_static_include]
Otherwise the main program is not different from an equivalent program using
the dynamic model. This feature makes it easy to develop the lexer in dynamic
mode and to switch to the static mode after the code has been stabilized.
The simple generator application shown above enables the integration of the
code generator into any existing build process. The following code snippet
provides the overall main function, highlighting the code to be changed.
[wc_static_main]
[important The generated code for the static lexer contains the token ids as
they have been assigned, either explicitly by the programmer or
implicitly during lexer construction. It is your responsibility
to make sure that all instances of a particular static lexer
type use exactly the same token ids. The constructor of the lexer
object has a second (default) parameter allowing it to designate a
starting token id to be used while assigning the ids to the token
definitions. The requirement above is fulfilled by default
as long as no `first_id` is specified during construction of the
static lexer instances.
]
[endsect]

View File

@@ -0,0 +1,57 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_tutorials __lex__ Tutorials Overview]
The __lex__ library implements several components on top of possibly different
lexer generator libraries. It exposes a pair of iterators, which, when
dereferenced, return a stream of tokens generated from the underlying character
stream. The generated tokens are based on the token definitions supplied by the
user.
Currently, __lex__ is built on top of Ben Hanson's excellent __lexertl__
library (which is a proposed Boost library). __lexertl__ provides the necessary
functionality to build state machines based on a set of supplied regular
expressions. But __lex__ is not restricted to be used with __lexertl__. We
expect it to be usable in conjunction with any other lexical scanner generator
library, all what needs to be implemented is a set of wrapper objects exposing a
well defined interface as described in this documentation.
[note For the sake of clarity all examples in this documentation assume
__lex__ to be used on top of __lexertl__.]
Building a lexer using __lex__ is highly configurable, where most of this
configuration is done at compile time. Almost all of the configurable
parameters have generally useful default values, allowing project startup to be
a easy and straightforward task. Here is a (non-complete) list of features you
can tweak to adjust the generated lexer instance to the actual needs:
* Select and customize the token type to be generated by the lexer instance.
* Select and customize the token value types the generated token instances will
be able to hold.
* Select the iterator type of the underlying input stream, which will be used
as the source for the character stream to tokenize.
* Customize the iterator type returned by the lexer to enable debug support,
special handling of certain input sequences, etc.
* Select the /dynamic/ or the /static/ runtime model for the lexical
analyzer.
Special care has been taken during the development of the library that
optimal code will be generated regardless of the configuration options
selected.
The series of tutorial examples of this section will guide you through some
common use cases helping to understand the big picture. The first two quick
start examples (__sec_lex_quickstart_1__ and __sec_lex_quickstart_2__)
introduce the __lex__ library while building two stand alone applications, not
being connected to or depending on any other part of __spirit__. The section
__sec_lex_quickstart_3__ demonstrates how to use a lexer in conjunction with a
parser (where obviously the parser is built using __qi__).
[endsect]

View File

@@ -0,0 +1,15 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_parsing Parsing using a Lexer]
[/ write about integration of lexer component with __qi__]
[/ write about iterator interface exposed by a __lex__ lexer]
[endsect]

View File

@@ -0,0 +1,117 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
This quick reference section is provided for convenience. You can use
this section as a sort of a "cheat-sheet" on the most commonly used Lex
components. It is not intended to be complete, but should give you an
easy way to recall a particular component without having to dig through
pages and pages of reference documentation.
[/////////////////////////////////////////////////////////////////////////////]
[section Common Notation]
[variablelist Notation
[[`L`] [Lexer type]]
[[`l, a, b, c, d`] [Lexer objects]]
[[`Iterator`] [The type of an iterator referring to the underlying
input sequence]]
[[`IdType`] [The token id type]]
[[`Context`] [The lexer components `Context` type]]
[[`ch`] [Character-class specific character (See __char_class_types__)]]
[[`Ch`] [Character-class specific character type (See __char_class_types__)]]
[[`str`] [Character-class specific string (See __char_class_types__)]]
[[`Str`] [Character-class specific string type (See __char_class_types__)]]
[[`Attrib`] [An attribute type]]
[[`fa`] [A semantic action function with a signature:
`void f(Iterator&, Iterator&, pass_flag&, Idtype&, Context&)`.]]
]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section:lexers Primitive Lexer Components]
[table
[[Expression] [Attribute] [Description]]
[[`ch`] [n/a] [Matches `ch`]]
[[`char_(ch)`] [n/a] [Matches `ch`]]
[[`str`] [n/a] [Matches regular expression `str`]]
[[`string(str)`] [n/a] [Matches regular expression `str`]]
[[`token_def<Attrib>`] [`Attrib`] [Matches the immediate argument]]
[[`a | b`] [n/a] [Matches any of the expressions `a` or `b`]]
[[`l[fa]`] [Attribute of `l`] [Call semantic action `fa` (after matching `l`).]]
]
[note The column /Attribute/ in the table above lists the parser attribute
exposed by the lexer component if it is used as a parser (see
__attribute__). A 'n/a' in this columns means the lexer component is not
usable as a parser.]
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section Semantic Actions]
Has the form:
l[f]
where `f` is a function with the signatures:
void f();
void f(Iterator&, Iterator&);
void f(Iterator&, Iterator&, pass_flag&);
void f(Iterator&, Iterator&, pass_flag&, Idtype&);
void f(Iterator&, Iterator&, pass_flag&, Idtype&, Context&);
You can use __boost_bind__ to bind member functions. For function
objects, the allowed signatures are:
void operator()(unused_type, unused_type, unused_type, unused_type, unused_type) const;
void operator()(Iterator&, Iterator&, unused_type, unused_type, unused_type) const;
void operator()(Iterator&, Iterator&, pass_flag&, unused_type, unused_type) const;
void operator()(Iterator&, Iterator&, pass_flag&, Idtype&, unused_type) const;
void operator()(Iterator&, Iterator&, pass_flag&, Idtype&, Context&) const;
The `unused_type` is used in the signatures above to signify 'don't
care'.
For more information see __lex_actions__.
[endsect]
[/////////////////////////////////////////////////////////////////////////////]
[section Phoenix]
__phoenix__ makes it easier to attach semantic actions. You just
inline your lambda expressions:
l[phoenix-lambda-expression]
__lex__ provides some __phoenix__ placeholders to access important
information from the `Context` that are otherwise difficult to extract.
[variablelist Spirit.Lex specific Phoenix placeholders
[[`_start, _end`] [Iterators pointing to the begin and the end of the
matched input sequence.]]
[[`_pass`] [Assign `lex::pass_flags::pass_fail` to `_pass` to force the current match to fail.]]
[[`_tokenid`] [The token id of the matched token.]]
[[`_val`] [The token value of the matched token.]]
[[`_state`] [The lexer state the token has been matched in.]]
[[`_eoi`] [Iterator referring to the current end of the input sequence.]]
]
[tip All of the placeholders in the list above (except `_eoi`) can be changed
from the inside of the semantic action allowing to modify the lexer
behavior. They are defined in the namespace `boost::spirit::lex`.]
For more information see __lex_actions__.
[endsect]

View File

@@ -0,0 +1,10 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section Token]
[endsect]

View File

@@ -0,0 +1,10 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:token Token Class]
[endsect]

View File

@@ -0,0 +1,11 @@
[/==============================================================================
Copyright (C) 2001-2011 Joel de Guzman
Copyright (C) 2001-2011 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:lexer_token_definition Ways to define Tokens]
[endsect]

View File

@@ -0,0 +1,76 @@
[/==============================================================================
Copyright (C) 2001-2011 Hartmut Kaiser
Copyright (C) 2001-2011 Joel de Guzman
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
===============================================================================/]
[section:primitives Token definition Primitives]
This module includes different primitives allowing you to create token definitions.
It includes `char_`, character literals, `string`, and string literals.
[heading Module Headers]
// forwards to <boost/spirit/home/lex/primitives.hpp>
#include <boost/spirit/include/lex_primitives.hpp>
Also, see __include_structure__.
[/////////////////////////////////////////////////////////////////////////////]
[section:char_tokendef Tokens Matching Single Characters]
[heading Description]
The character based token definitions described in this section are:
The `char_` creates token definitions matching single characters. The `char_`
token definition is associated `standard` encoding namespace. This is
needed when doing basic operations such as forcing lower or upper case and
dealing with character ranges.
[heading Header]
[heading Module Headers]
// forwards to <boost/spirit/home/lex/lexer/char_token_def.hpp>
#include <boost/spirit/include/lex_char_token_def.hpp>
Also, see __include_structure__.
[heading Namespace]
[table
[[Name]]
[[`lex::char_`]]
]
[heading Model of]
[:__primitive_lexer_concept__]
[variablelist Notation
[[`ch`] [Character-class specific character from `standard` character
set.]]
]
[heading Expression Semantics]
Semantics of an expression is defined only where it differs from, or is not
defined in __primitive_lexer_concept__.
[table
[[Expression] [Description]]
[[`ch`] [Create a token definition matching the character
literal `ch`. ]]
[[`lex::char_(ch)`] [Create a token definition matching the character
`ch`.]]
]
[heading Example]
[endsect] [/ char]
[endsect] [/ Module]

Some files were not shown because too many files have changed in this diff Show More