README.md           |   2 +-
 src/tokenizer.cpp   | 103 +++++++++++++++++++++++++++++++-
 src/tokenizer.hpp   |   8 ---
 test/standalone.cpp |  25 +++++++-
 4 files changed, 124 insertions(+), 14 deletions(-)