19.4 Life cycle mutators

Mutator function: int marpa_r_start_input (Marpa_Recognizer r)

When successful, does the following:

• Readies r to accept input.
• Completes the first Earley set. The ID of the first Earley set is 0, and it is located at earleme 0.
• Leaves the latest, current and furthest earlemes all at 0.
• Clears any events that were in the event queue before this method was called.
• If this method exhausts the parse, triggers a MARPA_EVENT_EXHAUSTED event. See Exhaustion.
• May trigger one or more MARPA_EVENT_SYMBOL_NULLED, MARPA_EVENT_SYMBOL_PREDICTED, or MARPA_EVENT_SYMBOL_EXPECTED events. See Events.

Return value: On success, a non-negative value, whose value is otherwise unspecified. On hard failure, -2.

Mutator function: int marpa_r_alternative (Marpa_Recognizer r, Marpa_Symbol_ID token_id, int value, int length)

The token_id argument must be the symbol ID of a terminal. The value argument is an integer that represents the “value” of the token, and which should not be zero. The length argument is the length of the token, which must be greater than zero.

On success, does the following, where current is the value of the current earleme before the call and furthest is the value of the furthest earleme before the call:

• Reads a new token into r. The symbol ID of the token will be token_id. The token will start at current and end at current+length.
• Sets the value of the furthest earleme to max(current+length,furthest).
• Leaves the values of the latest and current earlemes unchanged.

After recoverable failure, the following are the case:

• The tokens read into r are unchanged. Specifically, no new token has been read into r.
• The values of the latest, current and furthest earlemes are unchanged.

Libmarpa allows tokens to be ambiguous. Two tokens are ambiguous if they end at the same earleme location. If two tokens are ambiguous, Libmarpa will attempt to produce all the parses that include either of them.

Libmarpa allows tokens to overlap. Let the notation t@s-e indicate that token t starts at earleme s and ends at earleme e. Let t1@s1-e1 and t2@s2-e2 be two tokens such that s1<=s2. We say that t1 and t2 overlap iff e1>s2.

The value argument is not used inside Libmarpa — it is simply stored to be returned by the valuator as a convenience for the application. In applications where the token’s actual value is not an integer, it is expected that the application will use value as a “virtual” value, perhaps finding the actual value by using value to index an array. Some applications may prefer to track token values on their own, perhaps based on the earleme location and token_id, instead of using Libmarpa’s token values.

A value of 0 does not cause a failure, but it is reserved for unvalued symbols, a now-deprecated feature. See Valued and unvalued symbols.

Hard fails irrecoverably with MARPA_ERR_DUPLICATE_TOKEN if the token added would be a duplicate. Two tokens are duplicates iff all of the following are true:

• They would have the same start earleme. In other words, if marpa_r_alternative() attempts to read them while at the same current earleme.
• They have the same token_id.
• They have the same length.

If a token was not accepted because of its token ID, hard fails with the MARPA_ERR_UNEXPECTED_TOKEN_ID. This hard failure is fully recoverable so that, for example, the application may retry this method with different token IDs until it succeeds. These retries are efficient, and are quite useable as a parsing technique — so much so we have given the technique a name: the Ruby Slippers. The Ruby Slippers are used in several applications.

Return value: On success, MARPA_ERR_NONE. On hard failure, an error code other than MARPA_ERR_NONE. The hard failure for MARPA_ERR_UNEXPECTED_TOKEN_ID is fully recoverable.

Mutator function: int marpa_r_earleme_complete (Marpa_Recognizer r)

For the purposes of this method description, we define the following:

• current is the value of the current earleme before the call of marpa_r_earleme_complete.
• latest is the value of the latest earleme before the call of marpa_r_earleme_complete.
• An “expected” terminal is one expected at a current earleme, in the same sense that marpa_r_terminal_is_expected() determines if a terminal is “expected” at the current earleme. See marpa_r_terminals_expected().
• An “anticipated” terminal is one that was accepted by the marpa_r_alternative() to end at an earleme after the current earleme. An anticipated terminal will have length greater than one. “Anticipated” terminals only occur if the application is using an advanced model of input. See Advanced input models.

In this method description, we will frequently refer to parse exhaustion. Parse exhaustion is discussed in detail in its own section. See Exhaustion.

On success, does the final processing for the current earleme, including the following:

• Advances the current earleme, incrementing its value by 1. That is, sets the current earleme to current+1.
• If any token was accepted at current, creates a new Earley set, which will be the latest Earley set. After the call, the latest earleme will be equal to the new current earleme, current+1.
• If no token was accepted at current, no Earley set is created. After the call, the value of the latest earleme will be unchanged — that is, it will remain at latest. Success when no tokens were accepted at current can only occur if the application is using an advanced model of input. See Advanced input models.
• The value of the furthest earleme is never changed by a call to marpa_r_earleme_complete().
• Clears the event queue of any events that occured before this method was called.
• May trigger one or more MARPA_EVENT_SYMBOL_COMPLETED, MARPA_EVENT_SYMBOL_NULLED, MARPA_EVENT_SYMBOL_PREDICTED, or MARPA_EVENT_SYMBOL_EXPECTED events. See Events.
• If an application-settable threshold on the number of Earley items has been reached or exceeded, triggers a MARPA_EVENT_EARLEY_ITEM_THRESHOLD event. Often, the application will want to treat this event as if it were a ancestry-recoverable failure. See marpa_r_earley_item_warning_threshold_set().
• If the parse is exhausted, triggers a MARPA_EVENT_EXHAUSTED event. Exhaustion on success only occurs if no terminals are expected at the current earleme after the call to this method (that is, at current+1) and no terminals are anticipated after current+1.

On hard failure with the code MARPA_ERR_PARSE_EXHAUSTED, does the following:

• Leaves the current earleme at current. The current earleme will be the same as the furthest earleme.
• The value of the furthest earleme is never changed by a call to marpa_r_earleme_complete().
• Leaves the value of the latest earleme at latest. No new Earley set is created.
• Sets the parse exhausted, so that no more tokens will be accepted.
• Leaves the parse in a state where no terminals are expected or anticipated.
• Clears the event queue of any events that occured before the call to this method.
• Triggers a MARPA_EVENT_EXHAUSTED event and no others.
• Leaves valid any parses that were valid at the current or earlier earlemes. Processing with these can continue, and it for this reason that we consider hard failures with the code MARPA_ERR_PARSE_EXHAUSTED to be fully recoverable.

We note that exhaustion can occur when this method fails and when it succeeds. The distinction is that, on success, the call creates a new Earley set before becoming exhausted while, on failure, it becomes exhausted without creating a new Earley set.

This method is commonly called at the top of a loop. Almost all applications will want to check the return value and take special action in case of a value other than zero. If the value is greater than zero, an event will have occurred and almost all applications should react to MARPA_EVENT_EARLEY_ITEM_THRESHOLD events, as described above, and to unexpected events. If the value is less than zero, it may be due to an irrecoverable error, and only in very unusual circumstances will an application wish to ignore these.

How an application reacts to exhaustion will depend on the kind of parsing it is doing. See Exhaustion.

It is often up to the logic immediately around this method to detect EOP. See End of parse.

Return value: On success, the number of events in the event queue when the method call returns. On hard failure, -2. Hard failure with the code MARPA_ERR_PARSE_EXHAUSTED is fully recoverable.