Ocean of Awareness

Is Earley parsing fast enough?

"First we ask, what impact will our algorithm have on the parsing done in production compilers for existing programming languages? The answer is, practically none." -- Jay Earley's Ph.D thesis, p. 122.

In the above quote, the inventor of the Earley parsing algorithm poses a question. Is his algorithm fast enough for a production compiler? His answer is a stark "no".

This is the verdict on Earley's that you often hear repeated today, 45 years later. Earley's, it is said, has a too high a "constant factor". Verdicts tends to be repeated more often than examined. This particular verdict originates with the inventor himself. So perhaps it is not astonishing that many treat the dismissal of Earley's on grounds of speed to be as valid today as it was in 1968.

But in the past 45 years, computer technology has changed beyond recognition and researchers have made several significant improvements to Earley's. It is time to reopen this case.

What is a "constant factor"

The term "constant factor" here has a special meaning, one worth looking at carefully. Programmers talk about time efficiency in two ways: time complexity and speed.

Speed is simple: It's how fast the algorithm is against the clock. To make comparison easy, the clock can be an abstraction. The clock ticks could be, for example, weighted instructions on some convenient and mutually-agreed architecture.

By the time Earley was writing, programmers had discovered that simply comparing speeds, even on well-chosen abstract clocks, was not enough. Computers were improving very quickly. A speed result that was clearly significant when the comparison was made could quickly become unimportant. Researchers needed to talk about time efficiency in a way that made what they said as true decades later as on the day they said it. To do this, researchers created the idea of time complexity.

Time complexity is measured using several notations, but the most common is big-O notation. Here's the idea: Assume we are comparing two algorithms, Algorithm A and Algorithm B. Assume that algorithm A uses 42 weighted instructions for each input symbol. Assume that algorithm B uses 1792 weighted instructions for each input symbol. Where the count of input symbols is N, A's speed is 42*N, and B's is 1792*N. But the time complexity of both is the same: O(N). The big-O notation throws away the two "constant factors", 42 and 1792. Both are said to be "linear in N". (Or more often, just "linear".)

It often happens that algorithms we need to compare for time efficiency have different speeds, but the same time complexity. In practice, this usually this means we can treat them as having essentially the same time efficiency. But not always. It sometimes happens that this difference is relevant. When this happens, the rap against the slower algorithm is that it has a "high constant factor".

OK, about that high constant factor

What is the "constant factor" between Earley and the current favorite parsing algorithm, as a number? (My interest is practical, not historic, so I will be talking about Earley's as modernized by Aycock, Horspool, Leo and myself. But much of what I say applies to Earley's algorithm in general.)

What the current favorite parsing algorithm is can be an interesting question. When Earley wrote, it was hand-written recursive descent. The next year (1969) LALR parsing was invented, and the year after (1970) a tool that used it was introduced -- yacc. At points over the next decades, yacc chased both Earley's and recursive descent almost completely out of the textbooks. But as I have detailed elsewhere, yacc had serious problems. In 2006 things went full circle -- the industry's standard C compiler, GCC, replaced LALR with recursive descent.

So back to 1970. That year, Jay Earley wrote up his algorithm for "Communications of the ACM", and put a rough number on his "constant factor". He said that his algorithm was an "order of magnitude" slower than the current favorites -- a factor of 10. Earley suggested ways to lower this 10-handicap, and modern implementations have followed up on them and found others. But for this post, let's concede the factor of ten and throw in another. Let's say Earley's is 100 times slower than the current favorite, whatever that happens to be.

Moore's Law and beyond

Let's look at the handicap of 100 in the light of Moore's Law. Since 1968, computers have gotten a billion times faster -- nine orders of magnitude. Nine factors of ten. This means that today Earley's runs seven factors of ten faster than the current favorite algorithm did in 1968. Earley's is 10 million times as fast as the algorithm that was then considered practical.

Of course, our standard of "fast enough to be practical" also evolves. But it evolves a lot more slowly. Let's exaggerate and say that "practical" meant "takes an hour" in 1968, but that today we would demand that the same program take only a second. Do the arithmetic and you find that Earley's is now more than 2,000 times faster than it needs to be to be practical.

Bringing in Moore's Law is just the beginning. The handicap Jay Earley gave his algorithm is based on a straight comparison of CPU speeds. But parsing, in practical cases, involves I/O. And the "current favorite" needs to do as much I/O as Earley's. I/O overheads, and the accompanying context switches, swamp considerations of CPU speed, and that is more true today that it was in 1968. When an application is I/O bound, CPU is in effect free. Parsing may not be I/O bound in this sense, but neither is it one of those applications where the comparison can be made in raw CPU terms.

Finally, pipelining has changed the nature of the CPU overhead itself radically. In 1968, the time to run a series of CPU instructions varied linearly with the number of instructions. Today, that is no longer true, and the change favors strategies like Earley's, which require a higher instruction count, but achieve efficiency in other ways.

Achievable speed

So far, I've spoken in terms of theoretical speeds, not achievable ones. That is, I've assumed that both Earley's and the current favorite are producing their best speed, unimpeded by implementation considerations.

Earley, writing in 1968 and thinking of hand-written recursive descent, assumed that production compilers could be, and in practice usually would be, written by programmers with plenty of time to do careful and well-thought-out hand-optimization. After forty-five years of real-life experience, we know better.

In those widely used practical compilers and interpreters that rely on lots of procedural logic -- and these days that is almost all of them -- it is usually all the maintainers can do to keep the procedural logic correct. In all but a few cases, optimization is opportunistic, not systematic. Programmers have been exposed to the realities of parsing with large amounts of complex procedural logic, and hand-written recursive descent has acquired a reputation for being slow.

In theory, LALR based compilers are less dependent on procedural parsing and therefore easier to keep optimal. In practice they are as bad or worse. LALR parsers usually still need a considerable amount of procedural logic, but procedural logic is harder to write for LALR than it is for recursive descent.

Modern Earley parsing has a much easier time actually delivering its theoretical best speed in practice. Earley's is powerful enough, and in its modern version well-enough aware of the state of the parse, that procedural logic can be kept to minimum or eliminated. Most of the parsing is done by the mathematics at its core.

The math at Earley's core can be heavily optimized, and any optimization benefits all applications. Optimization of special-purpose procedural logic benefits only the application that uses that logic.

Other considerations

But you might say,

"A lot of interesting points, Jeffrey, but all things being equal, a factor of 10, or even what's left from a factor of ten once I/O, pipelining and implementation inefficiencies have all nibbled away at it, is still worth having. It may in a lot of instances not even be measurable, but why not grab it for the sake of the cases where it is?"

Which is a good point. The "implementation inefficiences" can be nasty enough that Earley's is in fact faster in raw terms, but let's assume that some cost in speed is still being paid for the use of Earley's. Why incur that cost?

Error diagnosis

The parsing algorithms currently favored, in their quest for efficiency, do not maintain full information about the state of the parse. This is fine when the source is 100% correct, but in practice an important function of a parser is to find and diagnose errors. When the parse fails, the current favorites often have little idea of why. An Earley parser knows the full state of the parse. This added knowledge can save a lot of programmer time.

Readability

The more that a parser does from the grammar, and the less procedural logic it uses, the more readable the code will be. This has a determining effect on maintainance costs and the software's ability to evolve over time.

Accuracy

Procedural logic can produce inaccuracy -- inability to describe or control the actual language begin parsed. Some parsers, particularly LALR and PEG, have a second major source of inaccuracy -- they use a precedence scheme for conflict resolution. In specific cases, this can work, but precedence-driven conflict resolution produces a language without a "clean" theoretical description.

The obvious problem with not knowing what language you are parsing is failure to parse correct source code. But another, more subtle, problem can be worse over the life cycle of a language ...

False positives

False positives are cases where the input is in error, and should be reported as such, but instead the result is what you wanted. This may sound like unexpected good news, but when a false positive does surface, it is quite possible that it cannot be fixed without breaking code that, while incorrect, does work. Over the life of a language, false positives are deadly. False positives produce buggy and poorly understood code which must be preserved and maintained forever.

Power

The modern Earley implementation can parse vast classes of grammar in linear time. These classes include all those currently in practical use.

Flexibility

Modern Earley implementations parse all context-free grammars in times that are, in practice, considered optimal. With other parsers, the class of grammars parsed is highly restricted, and there is usually a real danger that a new change will violate those restrictions. As mentioned, the favorite alternatives to Earley's make it hard to know exactly what language you are, in fact, parsing. A change can break one of these parsers without there being any indication. By comparison, syntax changes and extensions to Earley's grammars are carefree.

For more about Marpa

Above I've spoken of "modern Earley parsing", by which I've meant Earley parsing as amended and improved by the efforts of Aho, Horspool, Leo and myself. At the moment, the only implementation that contains all of these modernizations is Marpa.

Marpa's latest version is Marpa::R2, which is available on CPAN. Marpa's SLIF is a new interface, which represents a major increase in Marpa's "whipitupitude". The SLIF has tutorials here and here. Marpa has a web page, and of course it is the focus of my "Ocean of Awareness" blog.

Comments on this post can be sent to the Marpa's Google Group: marpa-parser@googlegroups.com

Marpa's SLIF now allows procedural parsing

Marpa's SLIF (scanless interface) allows an application to parse directly from any BNF grammar. Marpa parses vast classes of grammars in linear time, including all those classes currently in practical use. With its latest release, Marpa::R2's SLIF also allows an application to intermix its own custom lexing and parsing logic with Marpa's, and to switch back and forth between them. This means, among other things, that Marpa's SLIF can now do procedural parsing.

What is procedural parsing? Procedural parsing is parsing using ad hoc code in a procedural language. The opposite of procedural parsing is declarative parsing -- parsing driven by some kind of formal description of the grammar. Procedural parsing may be described as what you do when you've given up on your parsing algorithm. Dissatisfaction with parsing theory has left modern programmers accustomed to procedural parsing. And in fact some problems are best tackled with procedural parsing.

An example

One such problem is parsing Perl-style here-documents. Peter Stuifzand has tackled this using the just-released version of Marpa::R2. For those unfamiliar, Perl allows documents to be incorporated into its source files in line-oriented fashion as "here-documents". Here-documents can be used in expressions. The syntax to do this is very handy, if a little strange. For example,

say <<ENDA, <<ENDB, <<ENDC; say <<ENDD;
a
ENDA
b
ENDB
c
ENDC
d
ENDD

starts with a single line declaring four here-documents spread out over two say statements. The expressions of the form

<<ENDX

are here-document expressions. << is the heredoc operator. The string which follows it (in this example, ENDA, ENDB, etc.) is the heredoc terminator string -- the string that will signal end of body of the here-document. The body of the here-documents follow, in order, over the next eight lines. More details of here-document syntax, with examples, can be found in the Perl documentation.

All of this poses quite a challenge to a parser-lexer combination, which is one reason I chose it as an example -- to illustrate that the Marpa's SLIF support for procedural parsing can handle genuinely difficult cases. There are a few ways Marpa could approach this. The one Peter Stuifzand chose was to to read the here-document's body as the value of the terminator in each <<ENDX expression.

The strategy works this way: Marpa allows the application to mark certain lexemes as "pause" lexemes. Whenever a "pause" lexeme is encountered, Marpa's internal scanning stops, and control is handed over to the application. In this case, the application is set up to pause after every newline, and before the terminator in every here-document expression.

While reading the line containing the four here-document expressions, Marpa's SLIF pauses and resumes five times -- once for each here-document expression, then once for the final newline. Details can be found in compact form in the heavily commented code in this Github gist.

Marpa as a better procedural parser

So far I've talked in terms of Marpa "allowing" procedural parsing. In fact, there can be much more to it. Marpa can make procedural parsing easier and more accurate.

Marpa knows, at every point, which rules it is recognizing, and how far it is into them. Marpa also knows which new rules the grammar expects, and which terminals. The procedural parsing logic can consult this information to guide its decisions. Marpa can provide your procedural parsing logic with radar, as well as the option to use a very smart autopilot.

For more about Marpa

Comments on this post can be sent to the Marpa's Google Group: marpa-parser@googlegroups.com

What if languages were free?

In 1980, George Copeland wrote an article titled "What if Mass Storage were Free?". Costs of mass storage were showing signs that they might fall dramatically. Copeland, as a thought exercise, took this trend to its extreme. Among other things, he predicted that deletion would become unnecessary, and in fact, undesirable.

Copeland's thought experiment has proved prophetic. For many purposes, mass storage is treated as if it were free. For example, you probably retrieved this blog post from a server provided to me at no charge, in the hope that I might write and upload something interesting.

Until now languages were high-cost efforts. Worse, language projects ran a high risk of disappointment, up to and including total failure. I believe those days are coming to an end.

Small languages, shaped to the problem domain

What if whenever you needed a new language, poof, it was there? You would be encouraged to tackle each problem domain with a new language dedicated to dealing with that domain. Since each language is no larger than its problem domain, learning a language would be essentially the same as learning the problem domain. The incremental effort required to learn the language itself would head toward zero.

No more language bloat

Language bloat would end. Currently, the risk and cost of developing languages make it imperative to extend the ones we have. Free languages mean fewer reasons to add features to existing languages.

No more search for THE perfect language

No language is perfect for all tasks. But because the high cost of languages favors large, general-purpose languages, we are compelled to try for perfection anyway. Ironically, we are often making the language worse, and we know it.

A world full of perfect languages

An older sense of the word perfect is "having all the properties or qualities requisite to its nature and kind". The C language might be called perfect in this sense. C lacks a lot of features that are highly desirable in most contexts. But for programming that is portable and close to the hardware, the C language is perfect or close to it. If languages were free, this is the kind of perfection that we would seek -- languages precisely fitted to their domain, so that adding to them cannot make them better.

Moving toward free

My own effort to contribute to a fall in the cost of languages is the Marpa parser. Marpa produces a reasonable parser for every language you can write in BNF. If the BNF is for a grammar in any of the classes currently in practical use, the parser Marpa produces will have linear speed. In one case, using Marpa, a targeted language was written in less than an hour. More typically, Marpa reduce the time needed to create new languages to hours.

As one example of going from "impossible" to "easy", I have written a drop-in solution to an example in the Gang of Four book. The Gang of Four described a language and its interpretation, but they did not include a parser. Creating a parser to fit their example would have been impossibly hard when the Gang of Four wrote. Using Marpa, it is easy. The parser can be found in this earlier blog post.

Marpa's latest version is Marpa::R2, which is available on CPAN. Recently, it has gained immensely in "whipitupitude" with a new interface, which has tutorials here and here. Marpa has a web page, and of course it is the focus of my "Ocean of Awareness" blog.

Comments on this post can be sent to the Marpa's Google Group: marpa-parser@googlegroups.com

The Interpreter Design Pattern

The influential Design Patterns book lays out 23 patterns for programming. One of them, the Interpreter Pattern, is rarely used. Steve Yegge puts it a bit more strikingly -- he says that the book contains 22 patterns and a practical joke.

That sounds (and in fact is) negative, but elsewhere Yegge says that "[t]ragically, the only [Go4] pattern that can help code get smaller (Interpreter) is utterly ignored by programmers". (The Design Patterns book has four authors, and is often called the Gang of Four book, or Go4.)

In fact, under various names and definitions, the Interpreter Pattern and its close relatives and/or identical twins are widely cited, much argued and highly praised[1]. As they should be. Languages are the most powerful and flexible design pattern of all. A language can include all, and only, the concepts relevent to your domain. A language can allow you to relate them in all, and only, the appropriate ways. A language can identify errors with pinpoint precision, hide implementation details, allow invisible "drop-in" enhancements, etc., etc., etc.

In fact languages are so powerful and flexible, that their use is pretty much universal. The choice is not whether or not to use a language to solve the problem, but whether to use a general-purpose language, or a domain-specific language. Put another way, if you decide not to use a language targeted to your domain, it almost always means that you are choosing to use another language that is not specifically fitted to your domain.

Why then, is the Interpreter Pattern so little used? Why does Yegge call it a practical joke?

There's a problem

The problem with the Interpreter Pattern is that you must turn your language into an AST -- that is, you must parse it somehow. Simplifying the language can help here. But if the point is to be simple at the expense of power and flexibility, you might as well stick with the other 22 design patterns.

On the other hand, creating a parser for anything but the simplest languages has been a time-consuming effort, and one of a kind known for disappointing results. In fact, language development efforts run a real risk of total failure.

How did the Go4 deal with this? They defined the problem away. They stated that the parsing issue was separate from the Interpreter Pattern, which was limited to what you did with the AST once you'd somehow come up with one.

But AST's don't (so to speak) grow on trees. You have to get one from somewhere. In their example, the Go4 simply built an AST in their code, node by node. In doing this, they bypassed the BNF and the problem of parsing. But they also bypassed their language and the whole point of the Interpreter Pattern.

Which is why Yegge characterized the chapter as a practical joke. And why other programming techniques and patterns are almost always preferred to the Interpreter Pattern.

Finding that one missing piece

So that's how the Go4 left things. A potentially great programming technique, made almost useless because of a missing piece. There was no easy, general, and practical way to generate AST's.

Few expected that to change. I was more optimistic than most. In 2007 I embarked on a full-time project: to create a parser based on Earley's algorithm. I was sure that it would fulfill two of the criteria -- it would be easy to use, and it would be general. As for practical -- well, a lot of parsing problems are small, and a lot of applications don't require a lot of speed, and for these I expected the result to be good enough.

What I didn't realize was that all of the problems preventing Earley's from seeing real, practical use has already been solved in the academic literature. I was not alone in not having put the picture together. The people who had solved the problems had focused on two disjoint sets of issues, and were unaware of each other's work. In 1991, in the Netherlands, the mathematican Joop Leo had arrived at an astounding result -- he showed how to make Earley's run in linear time for LR-regular grammars. LR-regular is a vast class of grammars. It easily includes, as a proper subset, every class of grammar now in practical use -- regular expressions, PEG, recursive descent, the LALR on which yacc and bison are based, you name it. (For those into the math, LR-regular includes LR(k) for all k, and therefore LL(k), also for all k.)

Leo's mathematical approach did not address some nagging practical issues, foremost among them the handling of nullable rules and symbols. But ten years later in Canada, Aycock and Horspool focused on exactly these issues, and solved them. Aycock-Horspool seem to have been unaware of Leo's earlier result. The time complexity of the Aycock-Horspool algorithm was essentially that of Earley's original algorithm.

Because of Leo's work, for any grammar in any class currently in practical use, an Earley's parser could be fast. If only it could be combined with the approach of Aycock and Horspool, I realized, Leo's speeds could be available in an everyday programming tool.

In changing the Earley parse engine, Aycock-Horspool and Leo had branched off in different directions. It was not obvious that their approaches could be combined, much less how. And in fact, the combination of the two is not a simple algorithm. But it is fast, and the new Marpa parse engine makes full information about the state of the parse (rules recognized, symbols expected, etc.) available as it proceeds. This is very convenient for, among other things, error reporting.

Eureka and all that

The result is an algorithm which parses anything you can write in BNF and does it in times considered optimal in practice. Unlike recursive descent, you don't have to write out the parser -- Marpa generates a parser for you, from the BNF. It's the easy, "drop-in" solution that the Go4 needed and did not have. A reworking of the Go4 example, with the missing parser added, is in a previous blog post, and the code for the reworking is in a Github gist.

More about Marpa

Comments on this post can be sent to the Marpa's Google Group: marpa-parser@googlegroups.com

Notes

Note 1: For example, the Wikipedia article on DSL's; Eric Raymond discussing mini-languages; "Notable Design Patterns for Domain-Specific Languages", Diomidis Spinellis; and the c2.com wiki.

BNF to AST

The latest version of Marpa takes parsing "whipitupitude" one step further. You can now go straight from a BNF description of your language, and an input string, to an abstract syntax tree (AST).

To illustrate, I'll use an example from the Gang of Four's (Go4's) chapter on the Interpreter pattern. (It's pages 243-255 of the Design Patterns book.) The Go4 knew of no easy general way to go from BNF to AST, so they dealt with that part of the interpreter problem by punting -- they did not even try to parse the input string. Instead they constructed the BNF they'd just presented and constructed an AST directly in their code.

The reason the Go4 didn't know of an easy, generally-applicable way to parse their example was that there was none. Now there is. In this post, Marpa will take us quickly and easily from BNF to AST. (Full code for this post can be found in a Github gist.)

The Go4's example was a simple boolean expression language, whose primary input was

true and x or y and not x

Here, in full, is the BNF for an slight elaboration of the Go4 example. It is written in the DSL for Marpa's Scanless interface (SLIF DSL), and includes specifications for building the AST.

:default ::= action => ::array

:start ::= <boolean expression>
<boolean expression> ::=
       <variable> bless => variable
     | '1' bless => constant
     | '0' bless => constant
     | ('(') <boolean expression> (')') action => ::first bless => ::undef
    || ('not') <boolean expression> bless => not
    || <boolean expression> ('and') <boolean expression> bless => and
    || <boolean expression> ('or') <boolean expression> bless => or

<variable> ~ [[:alpha:]] <zero or more word characters>
<zero or more word characters> ~ [\w]*

:discard ~ whitespace
whitespace ~ [\s]+

This syntax should be fairly transparent. In previous posts I've given a tutorial, and a a mini-tutorial. And of course, the interface is documented.

For those skimming, here are a few quick comments on less-obvious features. To guide Marpa in building the AST, the BNF statements have action and bless adverbs. The bless adverbs indicate a Perl class into which the node should be blessed. This is convenient for using an object-oriented approach with the AST. The action adverb tells Marpa how to build the nodes. "action => ::array" means the result of the rule should be an array containing its child nodes. "action => ::first" means the result of the rule should just be its first child. Many of the child symbols, especially literal strings of a structural nature, are in parentheses. This makes them invisible to the semantics.

A :default pseudo-rule specifies the defaults -- in this case the "action => ::array" adverb setting. The :start pseudo-rule specified the start symbol. The :discard pseudo-rule indicates that whitespace is to be discarded.

The Go4 did not deal with precedence. In their example, the input string is fully parenthesized, even though its priorities are the standard ones. I've eliminated the parentheses, because the standard precedence is implemented in SLIF grammar. The double vertical bar ("||") is a "loosen" operator -- an alternative after "loosen" operator will be at a looser precedence than the one before. Alternatives separated by a single bar are at the same precedence.

Creating the AST

Creating the AST is simple. First, we use Marpa to turn the above DSL for boolean expressions into a parser. (We'd saved the SLIF DSL source in the string $rules.)

my $grammar = Marpa::R2::Scanless::G->new(
    {   bless_package => 'Boolean_Expression',
        source        => \$rules,
    }   
);

Next we define a closure that uses $grammar to turn BNF into AST's.

sub bnf_to_ast {
    my ($bnf) = @_;
    my $recce = Marpa::R2::Scanless::R->new( { grammar => $grammar } );
    $recce->read( \$bnf );
    my $value_ref = $recce->value();
    if ( not defined $value_ref ) {
        die "No parse for $bnf";
    }
    return ${$value_ref};
} ## end sub bnf_to_ast

Where $bnf is our input string, we run it as follows:

my $ast1 = bnf_to_ast($bnf);

The AST

If we use Data::Dumper to examine the AST,

say Data::Dumper::Dumper($ast1) if $verbose_flag;

we see this:

$VAR1 = bless( [
                 bless( [
                          bless( [
                                   'true'
                                 ], 'Boolean_Expression::variable' ),
                          bless( [
                                   'x'
                                 ], 'Boolean_Expression::variable' )
                        ], 'Boolean_Expression::and' ),
                 bless( [
                          bless( [
                                   'y'
                                 ], 'Boolean_Expression::variable' ),
                          bless( [
                                   bless( [
                                            'x'
                                          ], 'Boolean_Expression::variable' )
                                 ], 'Boolean_Expression::not' )
                        ], 'Boolean_Expression::and' )
               ], 'Boolean_Expression::or' );

Processing the AST

In their example, the Go4 processed their AST in several ways: straight evaluation, copying, and substitution of the occurrences of a variable in one boolean expression by another boolean expression. It is obvious that the AST above is the computational equivalent of the Go4's AST, but for the sake of completeness I carry out the same operations in the Github gist.

AST creation via Marpa's SLIF is self-hosting -- the SLIF DSL is parsed into an AST, and a parser created by interpreting the AST. The Marpa SLIF DSL source file in this post, that describes boolean expressions, was itself turned into an AST on its way to becoming a parser that turns boolean expressions into AST's.

Comments

Comments on this post can be sent to the Marpa Google Group: marpa-parser@googlegroups.com