init: add unit tests for tokenizer.cpp

Test: this unit test
Change-Id: Iae24f3bf7850dd4b298281f6608bce195096d60a
2 files changed