doc/langref.md    |   6 +-
 src/tokenizer.cpp | 271 +++++++++++++++++++++++++-------
 src/tokenizer.hpp |  12 ++
 3 files changed, 227 insertions(+), 62 deletions(-)