Reads data from a stream and returns it token by token. More...

#include <util/StreamTokenizer.hpp>

Public Types
typedef unsigned	token_t
	type for describing type of token

Public Member Functions
	StreamTokenizer (std::istream &inputStream)
	Constructs a new StreamTokenizer instance.
	StreamTokenizer::~StreamTokenizer ()
	Destructs a StreamTokenizer instance.
void	inputStream (std::istream &inputStream)
	Sets the input stream.
void	reset ()
	Resets all internal state (except syntax).
void	separatorChar (char c)
	Sets separator attribute for given character.
void	separatorChars (char firstChar, char lastChar)
	Sets separator attribute for given range of characters.
void	wordChar (char c)
	Sets word attribute for given character.
void	wordChars (char firstChar, char lastChar)
	Sets word attribute for given range of characters.
void	whitespaceChar (char c)
	Sets whitespace attribute for given character.
void	commentChar (char c)
	Sets comment attribute for given character.
void	resetSyntax ()
	Resets all characters to default attributes.
token_t	nextToken ()
	Reads next token from stream.
token_t	tokenType () const
	Gets the type of current token.
char	separatorToken () const
	Gets current separator token.
const char *	wordToken () const
	Gets pointer to current word token.
size_t	wordTokenSize () const
	Gets size of current word token.
bool	uintToken (unsigned &val) const
	Gets the current word token as integer.

Static Public Attributes
static const token_t	TT_NONE = 0
	invalid token
static const token_t	TT_EOF = 1
	end of file token
static const token_t	TT_SEPARATOR = 2
	separator token
static const token_t	TT_WORD = 3
	word token

Detailed Description

Reads data from a stream and returns it token by token.

The tokenization is controlled by attributes assigned to characters and can be changed while tokenization is in progress. For better speed all read operations on the stream are buffered.

Definition at line 23 of file StreamTokenizer.hpp.

Member Typedef Documentation

typedef unsigned bil::StreamTokenizer::token_t

type for describing type of token

Definition at line 132 of file StreamTokenizer.hpp.

Constructor & Destructor Documentation

StreamTokenizer::StreamTokenizer ( std::istream & inputStream )

explicit

Constructs a new StreamTokenizer instance.

Parameters:

inputStream Stream to read from.

Definition at line 21 of file StreamTokenizer.cpp.

bil::StreamTokenizer::StreamTokenizer::~StreamTokenizer ( )

Destructs a StreamTokenizer instance.

Member Function Documentation

void StreamTokenizer::inputStream ( std::istream & inputStream )

Sets the input stream.

Will call reset().

Parameters:

inputStream The input stream.

Definition at line 46 of file StreamTokenizer.cpp.

void StreamTokenizer::reset ( )

Resets all internal state (except syntax).

All internal buffers are flushed and the tokenization state is set back to default values. Next data will be read from current stream position.

Definition at line 53 of file StreamTokenizer.cpp.

void StreamTokenizer::separatorChar ( char c )

Sets separator attribute for given character.

A char of this attribute will be returned as one token of TT_SEPARATOR type.

Parameters:

c	The char which attribute is to be set.

Definition at line 63 of file StreamTokenizer.cpp.

void StreamTokenizer::separatorChars	(	char	firstChar,
		char	lastChar
	)

Sets separator attribute for given range of characters.

A char of this attribute will be returned as one token of TT_SEPARATOR type.

Parameters:

firstChar	First char in range which attribute is to be set.
lastChar	Last char in range which attribute is to be set.

Definition at line 69 of file StreamTokenizer.cpp.

void StreamTokenizer::wordChar ( char c )

Sets word attribute for given character.

Word chars make up words and so a whole consecutive sequence of word chars will be returned as one token of TT_WORD type.

Parameters:

c	The char which attribute is to be set.

Definition at line 76 of file StreamTokenizer.cpp.

void StreamTokenizer::wordChars	(	char	firstChar,
		char	lastChar
	)

Sets word attribute for given range of characters.

Word chars make up words and so a whole consecutive sequence of word chars will be returned as one token of TT_WORD type.

Parameters:

firstChar	First char in range which attribute is to be set.
lastChar	Last char in range which attribute is to be set.

Definition at line 82 of file StreamTokenizer.cpp.

void StreamTokenizer::whitespaceChar ( char c )

Sets whitespace attribute for given character.

Whitespace chars will be ignored and not returned as tokens.

Parameters:

c	The char which attribute is to be set.

Definition at line 89 of file StreamTokenizer.cpp.

void StreamTokenizer::commentChar ( char c )

Sets comment attribute for given character.

Comment chars and all following chars will be ignored until the next new line.

Parameters:

c	The char which attribute is to be set.

Definition at line 95 of file StreamTokenizer.cpp.

void StreamTokenizer::resetSyntax ( )

Resets all characters to default attributes.

By default all chars have got the separator attribute.

Definition at line 101 of file StreamTokenizer.cpp.

StreamTokenizer::token_t StreamTokenizer::nextToken ( )

Reads next token from stream.

The token type and the token itself are saved and available via tokenType(), separatorToken() and wordToken().

Returns:: The type of the read token.

Exceptions:

.

Definition at line 108 of file StreamTokenizer.cpp.

StreamTokenizer::token_t StreamTokenizer::tokenType ( ) const

Gets the type of current token.

Returns:: The token type.

Definition at line 164 of file StreamTokenizer.cpp.

char StreamTokenizer::separatorToken ( ) const

Gets current separator token.

Returns:: The token.

Definition at line 170 of file StreamTokenizer.cpp.

const char * StreamTokenizer::wordToken ( ) const

Gets pointer to current word token.

Returns:: The token.

Definition at line 176 of file StreamTokenizer.cpp.

size_t StreamTokenizer::wordTokenSize ( ) const

Gets size of current word token.

Returns:: The token size.

Definition at line 183 of file StreamTokenizer.cpp.

bool StreamTokenizer::uintToken ( unsigned & val ) const

Gets the current word token as integer.

Parameters:

val	Reference to variable to hold value.

Returns:: True, if conversion suceeded; false otherwise.

Definition at line 189 of file StreamTokenizer.cpp.

Member Data Documentation

const token_t bil::StreamTokenizer::TT_NONE = 0

static

invalid token

Definition at line 135 of file StreamTokenizer.hpp.

const token_t bil::StreamTokenizer::TT_EOF = 1

static

end of file token

Definition at line 138 of file StreamTokenizer.hpp.

const token_t bil::StreamTokenizer::TT_SEPARATOR = 2

static

separator token

Definition at line 141 of file StreamTokenizer.hpp.

const token_t bil::StreamTokenizer::TT_WORD = 3

static

word token

Definition at line 144 of file StreamTokenizer.hpp.

The documentation for this class was generated from the following files:

util/StreamTokenizer.hpp
util/StreamTokenizer.cpp

Public Types

Public Member Functions

Static Public Attributes

Detailed Description

Member Typedef Documentation

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation