2005/5/9

--
down ACDK
down API Docs
down Namespaces
down acdk
down acdk::io
down StreamTokenizer
  down members
  down Java (java.sun.com)
See also:
  down StreamTokenizer.cpp
  down StreamTokenizer.h

(c) 2005 by
Roger Rene Kommer

Kindly hosted at:
SourceForge.net Logo

     
 

class StreamTokenizer
artefaktur
Modules   Namespace List   Class Hierarchy   Alphabetical List   Compound List   Namespace Members   Compound Members   Related Pages  

acdk::io::StreamTokenizer Class Reference

#include <StreamTokenizer.h>

Inheritance diagram for acdk::io::StreamTokenizer:

acdk::lang::Object acdk::lang::ObjectBase acdk::lang::dmi::StdDispatch acdk::lisp::LispTokenizer List of all members.

Detailed Description

Parses a character stream to token.

API: ACDK<br/>

Author:
Roger Rene Kommer (mailto:kommer@artefaktur.com)
Version:
Revision
1.26
Date:
Date
2005/04/09 19:26:45


Public Member Functions

 StreamTokenizer (IN(RCharReader) reader)
virtual ~StreamTokenizer ()
virtual void commentChar (int ch)
virtual void eolIsSignificant (bool flag)
virtual int lineno ()
virtual void lowerCaseMode (bool fl)
virtual int nextToken ()
virtual void ordinaryChar (int ch)
virtual void ordinaryChars (int low, int hi)
virtual void parseNumbers ()
virtual void pushBack ()
virtual void quoteChar (int ch)
virtual void resetSyntax ()
virtual RString toString ()
 returns current token for logging

RString toCode ()
 returns current token as original code

virtual void whitespaceChars (int low, int hi)
virtual void wordChars (int low, int hi)
virtual RString getDeviceName ()
virtual void skipLine ()
 reading the rest of current line, whithout parsing it.

bool wantWhiteSpace (bool b)
bool wantNewline (bool b)
bool wantComments (bool b)
 want parsed C/C++ comments return as token or discarge has no effect if parseCCComments() is false

void parseCCComments (bool b)
 should C/C++ comments parse

bool parseCCComments ()
void readCxxIdentifier (bool b)
bool readCxxIdentifier ()
void readOperator (bool b)
bool readOperator ()
bool readNumberAsString ()
void readNumberAsString (bool b)
int read ()
 API: Extended.

void unread (int c)
void unread (IN(RString) str)
void pushBack (int typ, IN(RString) stringval)
bool eof ()
RString getStreamPos (bool withXPos=true)
 For Debugging output returns the current positition of the stream.

CharStreamPos getCharStreamPos ()
void getCharStreamPos (OUT(int) charPos, OUT(int) linePos, OUT(int) columnPos)
RString lastReaded ()
acdk::lang::dmi::RDmiObject getCurrentToken ()
 return the current token as ScriptVar In case of number values return as smallest possible type

acdk::lang::dmi::ScriptVar getCurrentSvToken ()

Static Public Member Functions

RString toCode (int tk, IN(acdk::lang::dmi::ScriptVar) sv)
int mapCEscapedChar (int nc)
 maps a escaped character to the corresponding decoded character following C enconding '
' becomes new line '' a tab space '' x


Public Attributes

RNumber nval
RString sval
uc2char cval
int ttype

Static Public Attributes

const int TT_EOF
const int TT_EOL
const int TT_NUMBER
const int TT_WORD
const int TT_STRING
 a quoted String, following the c/java notation

const int TT_QCHAR
 a quoted character in the c/java notation

const int TT_CCOMMENT
 a comment, following the C-Notation.

const int TT_CXXCOMMENT
 a comment, beginning with 2 slashes until the end of line.

const int TT_WS
 a sequense of whitespaces.

const int TT_OPERATOR

Protected Member Functions

RString _readCxxComment ()
RString _readCComment ()
RString _readCString ()
bool _readCChar ()
 read a quoted character

RString _readUnescapedCString ()
RString _readWhiteSpaces ()
bool _isWhiteSpace (int c)
bool _readNumber ()
bool _readIdentifier ()

Protected Attributes

RPushbackCharReader _in
RLineNumberCharReader _lineReader
bool _wantWS
 Tokenizer should return white spaces.

bool _wantComments
 return parsed comments (C/C++-Comments) or just throw it away sval contains the comment text without the comment chars

bool _wantNL
 if true, a new line character will return seperated from WS, otherwise, it will return embedded in whitespaces.

bool _readCxxIdentifier
 if true read acdk::lang::String as identifier

bool _parseOperator
 parse standard operator and return TT_OPERATOR if not set, just return the standard character

bool _eof
 true if unlying stream is return -1

bool _readNumberAsString
 read numbers as strings, not as number

bool _parseCCComments
 parse C/C++ comments

Constructor & Destructor Documentation

acdk::io::StreamTokenizer::StreamTokenizer (  IN(RCharReader)  reader  ) 
 

virtual acdk::io::StreamTokenizer::~StreamTokenizer (   )  [virtual]
 

Member Function Documentation

bool acdk::io::StreamTokenizer::_isWhiteSpace (  int  c  )  [protected]
 

bool acdk::io::StreamTokenizer::_readCChar (   )  [protected]
 

read a quoted character

RString acdk::io::StreamTokenizer::_readCComment (   )  [protected]
 

RString acdk::io::StreamTokenizer::_readCString (   )  [protected]
 

RString acdk::io::StreamTokenizer::_readCxxComment (   )  [protected]
 

bool acdk::io::StreamTokenizer::_readIdentifier (   )  [protected]
 

Reimplemented in acdk::lisp::LispTokenizer.

bool acdk::io::StreamTokenizer::_readNumber (   )  [protected]
 

RString acdk::io::StreamTokenizer::_readUnescapedCString (   )  [protected]
 

RString acdk::io::StreamTokenizer::_readWhiteSpaces (   )  [protected]
 

virtual void acdk::io::StreamTokenizer::commentChar (  int  ch  )  [virtual]
 

bool acdk::io::StreamTokenizer::eof (   )  [inline]
 

Reimplemented in acdk::lisp::LispTokenizer.

virtual void acdk::io::StreamTokenizer::eolIsSignificant (  bool  flag  )  [virtual]
 

void acdk::io::StreamTokenizer::getCharStreamPos (  OUT(int)  charPos,
OUT(int)  linePos,
OUT(int)  columnPos
)  [inline]
 

CharStreamPos acdk::io::StreamTokenizer::getCharStreamPos (   ) 
 

acdk::lang::dmi::ScriptVar acdk::io::StreamTokenizer::getCurrentSvToken (   ) 
 

acdk::lang::dmi::RDmiObject acdk::io::StreamTokenizer::getCurrentToken (   )  [inline]
 

return the current token as ScriptVar In case of number values return as smallest possible type

virtual RString acdk::io::StreamTokenizer::getDeviceName (   )  [virtual]
 

RString acdk::io::StreamTokenizer::getStreamPos (  bool  withXPos = true  ) 
 

For Debugging output returns the current positition of the stream.

:,

RString acdk::io::StreamTokenizer::lastReaded (   ) 
 

virtual int acdk::io::StreamTokenizer::lineno (   )  [virtual]
 

virtual void acdk::io::StreamTokenizer::lowerCaseMode (  bool  fl  )  [virtual]
 

int acdk::io::StreamTokenizer::mapCEscapedChar (  int  nc  )  [static]
 

maps a escaped character to the corresponding decoded character following C enconding '
' becomes new line '' a tab space '' x

virtual int acdk::io::StreamTokenizer::nextToken (   )  [virtual]
 

Reimplemented in acdk::lisp::LispTokenizer.

virtual void acdk::io::StreamTokenizer::ordinaryChar (  int  ch  )  [virtual]
 

virtual void acdk::io::StreamTokenizer::ordinaryChars (  int  low,
int  hi
)  [virtual]
 

bool acdk::io::StreamTokenizer::parseCCComments (   )  [inline]
 

void acdk::io::StreamTokenizer::parseCCComments (  bool  b  )  [inline]
 

should C/C++ comments parse

virtual void acdk::io::StreamTokenizer::parseNumbers (   )  [virtual]
 

void acdk::io::StreamTokenizer::pushBack (  int  typ,
IN(RString)  stringval
) 
 

virtual void acdk::io::StreamTokenizer::pushBack (   )  [virtual]
 

virtual void acdk::io::StreamTokenizer::quoteChar (  int  ch  )  [virtual]
 

int acdk::io::StreamTokenizer::read (   ) 
 

API: Extended.

bool acdk::io::StreamTokenizer::readCxxIdentifier (   )  [inline]
 

void acdk::io::StreamTokenizer::readCxxIdentifier (  bool  b  )  [inline]
 

void acdk::io::StreamTokenizer::readNumberAsString (  bool  b  )  [inline]
 

bool acdk::io::StreamTokenizer::readNumberAsString (   )  [inline]
 

bool acdk::io::StreamTokenizer::readOperator (   )  [inline]
 

void acdk::io::StreamTokenizer::readOperator (  bool  b  )  [inline]
 

virtual void acdk::io::StreamTokenizer::resetSyntax (   )  [virtual]
 

virtual void acdk::io::StreamTokenizer::skipLine (   )  [virtual]
 

reading the rest of current line, whithout parsing it.

API: extended

RString acdk::io::StreamTokenizer::toCode (  int  tk,
IN(acdk::lang::dmi::ScriptVar)  sv
)  [static]
 

RString acdk::io::StreamTokenizer::toCode (   ) 
 

returns current token as original code

virtual RString acdk::io::StreamTokenizer::toString (   )  [virtual]
 

returns current token for logging

Reimplemented from acdk::lang::Object.

void acdk::io::StreamTokenizer::unread (  IN(RString)  str  ) 
 

void acdk::io::StreamTokenizer::unread (  int  c  ) 
 

bool acdk::io::StreamTokenizer::wantComments (  bool  b  )  [inline]
 

want parsed C/C++ comments return as token or discarge has no effect if parseCCComments() is false

bool acdk::io::StreamTokenizer::wantNewline (  bool  b  )  [inline]
 

bool acdk::io::StreamTokenizer::wantWhiteSpace (  bool  b  )  [inline]
 

virtual void acdk::io::StreamTokenizer::whitespaceChars (  int  low,
int  hi
)  [virtual]
 

virtual void acdk::io::StreamTokenizer::wordChars (  int  low,
int  hi
)  [virtual]
 

Member Data Documentation

bool acdk::io::StreamTokenizer::_eof [protected]
 

true if unlying stream is return -1

RPushbackCharReader acdk::io::StreamTokenizer::_in [protected]
 

RLineNumberCharReader acdk::io::StreamTokenizer::_lineReader [protected]
 

bool acdk::io::StreamTokenizer::_parseCCComments [protected]
 

parse C/C++ comments

bool acdk::io::StreamTokenizer::_parseOperator [protected]
 

parse standard operator and return TT_OPERATOR if not set, just return the standard character

bool acdk::io::StreamTokenizer::_readCxxIdentifier [protected]
 

if true read acdk::lang::String as identifier

bool acdk::io::StreamTokenizer::_readNumberAsString [protected]
 

read numbers as strings, not as number

bool acdk::io::StreamTokenizer::_wantComments [protected]
 

return parsed comments (C/C++-Comments) or just throw it away sval contains the comment text without the comment chars

bool acdk::io::StreamTokenizer::_wantNL [protected]
 

if true, a new line character will return seperated from WS, otherwise, it will return embedded in whitespaces.

bool acdk::io::StreamTokenizer::_wantWS [protected]
 

Tokenizer should return white spaces.

uc2char acdk::io::StreamTokenizer::cval
 

RNumber acdk::io::StreamTokenizer::nval
 

RString acdk::io::StreamTokenizer::sval
 

const int acdk::io::StreamTokenizer::TT_CCOMMENT [static]
 

a comment, following the C-Notation.

By default this token will not be returned. Use wantComments() to tongle this value

const int acdk::io::StreamTokenizer::TT_CXXCOMMENT [static]
 

a comment, beginning with 2 slashes until the end of line.

By default this token will not be returned. Use wantComments() to tongle this value

const int acdk::io::StreamTokenizer::TT_EOF [static]
 

const int acdk::io::StreamTokenizer::TT_EOL [static]
 

const int acdk::io::StreamTokenizer::TT_NUMBER [static]
 

const int acdk::io::StreamTokenizer::TT_OPERATOR [static]
 

const int acdk::io::StreamTokenizer::TT_QCHAR [static]
 

a quoted character in the c/java notation

const int acdk::io::StreamTokenizer::TT_STRING [static]
 

a quoted String, following the c/java notation

const int acdk::io::StreamTokenizer::TT_WORD [static]
 

const int acdk::io::StreamTokenizer::TT_WS [static]
 

a sequense of whitespaces.

Readed whitespace string can be found in sval. Only returns this token, if set with wantWhiteSpace(bool b);

int acdk::io::StreamTokenizer::ttype