9.3 Other ways of determining EOP

Occasionally, an exhaustion-hating application may not know the length of its input in advance. Since these applications will not know from the length of the input, or from exhaustion, that they are at end of input, they will need some other way of determining this. One way they may do this is with a MARPA_EVENT_SYMBOL_COMPLETED event.

An example of such an application that has seen much use is the Marpa::R2 lexer. The Marpa::R2 lexer looks for one of a set of “lexemes”, seeking the longest one. It does this by declaring a MARPA_EVENT_SYMBOL_COMPLETED event for each lexeme, and recording the most recent location at which one of these completion events occurs. The Marpa::R2 lexer stops looking for the longest lexeme on parse exhaustion, and at that point treats the parse location of the most recent MARPA_EVENT_SYMBOL_COMPLETED event as the EOP. If no MARPA_EVENT_SYMBOL_COMPLETED event occurred, the Marpa::R2 lexer failed to find a lexeme, and reports a lexing error.