Parsing v-type sequences

In the normal use of Boost PP, data is passed as arguments to a macro in discrete units so that each parameter expects a single data type. A typical macro might be:

#define AMACRO(anumber,atuple,anidentifier) someoutput

where the 'atuple', having the form of ( data1, data2, data3 ), itself may contain different data types of elements.

This is the standard macro design and internally it is the easiest way to pass macro data back and forth. The Boost PP library has a rich set of functionality to deal with all of its high-level data types and variadic data also offers another alternative to representing data with its own simpler functionality.

Occasionally designers of macros, especially for the use of others programmers within a particular library, have expressed the need for a macro parameter to allow a more C/C++ like syntax where a single parameter might mimic a C++ function-call or a C-like type modification syntax, or some other more complicated construct. Something along the lines of:

areturn afunction ( aparameter1, aparameter2, aparameter3 )

( type ) data

etc. etc.

In other words, from a syntactical level when designing possible macro input, is it possible to design parameter data to look more like C/C++ when macros are used in a library and still do a certain amount of preprocessor metaprogramming with such mixed token input ?

VMD has functionality which allows more than one type of preprocessing token, excluding an 'empty' token which always refers to entire input, to be part of a single parameter of input data, as long as all the top-level data of a parameter is of different v-types. What this means is that if some input consists of a sequence of v-types it is possible to extract the data for each v-type in the sequence.

In practicality what this means is that, given the examples just above, if 'areturn', 'afunction', and 'data' are v-identifiers it would be possible to parse either of the two inputs above so that one could identify the different v-types involved and do preprocessor metaprogramming based on those results.

Before I give an explanation of how this is done using VMD functionality I would like to make two points:

I am offering the possibility of more various syntaxes for the preprocessor metaprogrammer but I am aware that this can be easily abused. In general keeping things simple is usually better than making things overly complicated when it comes to the syntactical side of things in a computer language. A macro parameter syntactical possibility has to be understandable to be used.
Using VMD to parse the individual types of a single parameter takes more preprocessing time than functionality offered with Boost PP data types, because a good deal of it is based on preprocessor recursion techniques.

Using VMD to parse input

We have seen how VMD can test input to a macro for a particular v-type. In these cases the input must be entirely of that particular type in order to return non-zero as opposed to 0.

VMD also has functionality to extract from input a particular v-type at the beginning of the input along with the remaining preprocessing tokens of that input. What this means is that given some input you can, as an example, extract the beginning tuple as well as the rest of the input. By recursively applying this technique to the rest of the input you can parse the input for its top-level data.

The one constraint using this technique is that the top-level input must consist of v-types, in other words preprocessor tokens which VMD understands. Therefore if your data is one of the examples above, you will be successful in using VMD. However if your preprocessor data takes the form of:

&name identifier ( param )

identifier "string literal"

identifier + number

identifier += 4.3

etc. etc.

you will not be able to parse the data using VMD since '&', "string literal", '+', '+=', and "4.3" are preprocessor tokens which are not top-level v-types and therefore VMD cannot handle them at the parsing level. You can still of course pass such data as preprocessing input to macros but you cannot use VMD to recognize the parts of such data.

This is similar to the fact that VMD cannot tell you what type preprocessor data is as a whole, using a VMD identifying macro, if the type is not one that VMD can handle.

On the other hand you can still use VMD to parse such tokens in the input if you use Boost PP data types as top-level v-types to do so. Such as:

( &name ) identifier ( param )

identifier ( "string literal" )

identifier ( + ) number

identifier ( += ) 4 ( . ) 3

I will be calling such input data, which consists of more than one v-type, by the term of a 'v-sequence'.

The succeeding topics explain the VMD functionality for parsing a v-sequence for each individual v-type.

Parsing v-identifiers

The macro BOOST_VMD_IDENTIFIER looks for a v-identifier at the beginning of a v-sequence. Just like BOOST_VMD_IS_IDENTIFIER it takes as mandatory parameters a v-sequence as the first parameter and a v-key as the second parameter. It returns a tuple of two elements. The first tuple element is the same return value as BOOST_VMD_IS_IDENTIFIER; an index, starting with 1, of the v-identifier corresponding to its v-key or 0 if no v-identifier is found. The second tuple element is the rest of the v-sequence if a v-identifier is found or an empty element if a v-identifier is not found. Since a v-identifier can be found and the rest of the v-sequence can be empty, one always needs to check the first tuple element to determine if the v-identifier was found at the beginning of the v-sequence or not.

Usage

To use the BOOST_VMD_IDENTIFIER macro either include the general header:

#include <boost/vmd/vmd.hpp>

or include the specific header:

#include <boost/vmd/identifier.hpp>

Simple v-sequences

A beginning v-identifier in a v-sequence can always be parsed using the mandatory parameters if the v-identifier is followed by:

tuple ...
v-number tuple ...
v-number end-of_sequence
end-of-sequence

A 'tuple' here refers to any pair of parenthesis, such as an array, a list, or the beginning of a seq.

As we will see more complicated v-sequences beginning with a v-identifier can be parsed using the optional parameters of BOOST_VMD_IDENTIFIER. These optional parameters will be discussed later.

Some examples:

#define BOOST_VMD_MAP_MYLIB_KEY1_CIRCLE
#define BOOST_VMD_MAP_MYLIB_KEY2_SQUARE
#define BOOST_VMD_MAP_MYLIB_KEY3_RECTANGLE
#define BOOST_VMD_MAP_MYLIB_KEY4_TRIANGLE
#define BOOST_VMD_MAP_MYLIB_KEY5_SHAPE

#define SOME_INPUT1 CIRCLE (atuple_elem1,atuple_elem2) any_vtypes

BOOST_VMD_IDENTIFIER
  (
  SOME_INPUT1 /* v-sequence */,
  MYLIB_KEY1_ /* CIRCLE, key which matches */
  )

expands to (1,(atuple_elem1,atuple_elem2) any_vtypes)

#define SOME_INPUT2 SQUARE 147 (atuple_elem1,atuple_elem2) any_vtypes

BOOST_VMD_IDENTIFIER
  (
  SOME_INPUT2 /* v-sequence */,
  (MYLIB_KEY1_ /* CIRCLE */,MYLIB_KEY2_ /* SQUARE, key which matches */)
  )

expands to (2,147 (atuple_elem1,atuple_elem2) any_vtypes)

#define SOME_INPUT3 RECTANGLE 33

BOOST_VMD_IDENTIFIER
  (
  SOME_INPUT3 /* v-sequence */,
  MYLIB_KEY3_ /* RECTANGLE, key which matches */
  )

expands to (1,33)

#define SOME_INPUT4 TRIANGLE

BOOST_VMD_IDENTIFIER
  (
  SOME_INPUT4 /* v-sequence */,
  (MYLIB_KEY1_ /* CIRCLE */,MYLIB_KEY2_ /* SQUARE */,MYLIB_KEY3_ /* TRIANGLE, key which matches */)
  )

expands to (3,)

#define SOME_INPUT5 AVIDENT any_vtypes

BOOST_VMD_IDENTIFIER
  (
  SOME_INPUT5 /* v-sequence */,
  MYLIB_KEY5_ /* SHAPE, key does not match */
  )

expands to (0,) because the v-identifier was not found at the beginning of the input.