jET
1.0alpha5


prv.doman.util
Class Tokenizer

java.lang.Object
  |
  +--prv.doman.util.Tokenizer

public class Tokenizer
extends java.lang.Object

The stream tokenizer gets a stream and parses it into tokens.

There are some predefined tokens:

and regular words.

Tokenizer may ignore slash slash comments (like in C++).

Using

This is an example of using this class.

     Tokenizer t = new Tokenizer(new FileReader("note.txt"));
     while (t.hasMoreTokens())
     {
        Token token = t.nextToken();
        ...
     }
   

Caution!

This class is not optimized.

See Also:
Token

Constructor Summary
Tokenizer(java.io.Reader in)
          Creates tokenizer to tokenize reader input.
Tokenizer(java.lang.String source)
          Creates tokenizer to tokenize string.
 
Method Summary
 boolean getIgnoreSlashSlash()
          Gets slash slash comments ingoration flag.
 int getLine()
          Returns current line number.
 boolean hasMoreTokens()
          Returns true, if in the input is at least one token to read.
 Token nextToken()
          Gets next token from input and returns it.
 void setIgnoreSlashSlash(boolean b)
          Sets slash slash comments ignoration.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

Tokenizer

public Tokenizer(java.io.Reader in)
Creates tokenizer to tokenize reader input. After that tokenizer will not ignore cases and slash, slash comments.

It's much better to wrap input reader with buffering.

Parameters:
in - Input to tokenize.

Tokenizer

public Tokenizer(java.lang.String source)
Creates tokenizer to tokenize string.

Parameters:
source - String to tokenize.
Method Detail

hasMoreTokens

public boolean hasMoreTokens()
                      throws java.io.IOException
Returns true, if in the input is at least one token to read.

Returns:
True, if in the input is at least one token to read.
Throws:
java.io.IOException - When some error occured.

nextToken

public Token nextToken()
                throws java.io.IOException
Gets next token from input and returns it.

Returns:
The next token.
Throws:
java.io.IOException - When an error occured.

getLine

public int getLine()
Returns current line number.

Returns:
The line number.

setIgnoreSlashSlash

public void setIgnoreSlashSlash(boolean b)
Sets slash slash comments ignoration. Slash slash comments begin with two slashes // and end at the end of line.

Parameters:
b - If true, slash slash comments will be ignored.

getIgnoreSlashSlash

public boolean getIgnoreSlashSlash()
Gets slash slash comments ingoration flag.

Returns:
True means that tokenizer ignores slash slash comments.

jET
1.0alpha5