Alternative Byte Sources for the Lexer

[sebastianriese]

Currently the generated lexers can only open files by name and the files must be mmap-able.

Therefore it would be nice to be capable of lexing from other sources, such as strings, sockets, pipes, stdin etc. (On 32bit machines there may also be a problem when mmap-ing very large files due to limited address space).

While lexing from strings is trivially added (just the constructor of the Lexer class has to be adapted), lexing from other sources is an interesting task, especially if the lookahead should be limited (compare the flex interactive mode).

Additionally a stack of input files would be nice to enable inclusion of source files.

[sebastianriese]

Partially solved: We now can parse from strings and non-mmappable files, which are slurped (this behavious is obviously non-optimal in many cases and non-interactive).

[sebastianriese]

The switch to an InputBuffer which wraps the file for reading makes easy extension of byte-sources possible. Until now only string input and whole file input (via mmap or that is not possible by slurping the file). Other Strategies should be implemented.