tree f533e0cccaec4631dd1d37f5f6332122bb7ce418
parent fac3bf35af412292e1a955510095c4234dc0fd53
author Lee Campbell <leecam@google.com> 1438273631 -0700
committer Lee Campbell <leecam@google.com> 1438281917 +0000

init: Add C++ tokenizer.

Adds a C++ tokenizer along with unit tests.

This tokenizer will replace the current C implementation
which does a poor job of keeping track of pointers.

This CL is a prerequisite for up coming changes to
the parser. This CL does not wire up this tokenizer and
changes no exsiting code. All that builds is the unit tests.

Change-Id: Iec3740bce7153640adc5e5bbdc57e644cedf0038
TEST: Unit tests all pass. No leaks under valgrind
BUG: 22843198