Lexing

Lexical analysis (or "lexing") is the process of converting a text buffer into a sequence of tokens which uniquely identify the building blocks of a custom language. For example, a lexer can convert the C# expression age = 42; into the following stream of tokens:

A parser can then analyse this stream of tokens, looking for known patterns, and build a parse tree that identifies an assignment operation, with an identifier and an constant expression.

This section looks at how ReSharper expects to be able to create a lexer via a lexer factory, and the lexer itself must implement the ILexer interface, and use text buffers to access the text block to analyse. This section also takes a look at how to use the CsLex SDK tool to tokenise a custom language.

Finally, this section takes a look at a couple of useful features ReSharper uses while parsing - lexer utility methods, filtering lexers, and caching lexers.