RNGSGLR: Generalization of the Context-Aware Scanning Architecture for All Character-Level Context-Free Languages

Abstract

The limitations of traditional parsing architecture are well known. Even when paired with parsing methods that accept all context-free grammars (CFGs), the resulting combination for any given CFG accepts only a limited subset of corresponding character-level context-free languages (CFL). We present a novel scanner-based architecture that for any given CFG accepts all corresponding character-level CFLs. It can directly parse all possible specifications consisting of a grammar and regular definitions. The architecture is based on right-nulled generalized LR (RNGLR) parsing and is a generalization of the context-aware scanning architecture. Our architecture does not require any disambiguation rules to resolve lexical conflicts, it conceptually has an unbounded parser and scanner lookahead and it is streaming. The added robustness and flexibility allow for easier grammar development and modification.

1. Introduction

The traditional parsing architecture is based on the assumption that programming languages are akin to natural languages. The input string of characters is decomposed into a string of words, which then form sentences. The analogy is sound, which justifies the division of the process into two phases, scanning and parsing [1]. Traditionally, scanning is expected to be deterministic and context-independent, which contrasts with the fact that scanning is heavily non-deterministic for virtually all programming languages. The non-determinism is resolved using ad hoc disambiguation rules, such as the longest match, priority, scanner lookahead, disambiguation functions, symbol tables, and/or modifications to the grammar [2,3]. The crux of the problem is that, even if the parsing method accepts all context-free grammars (CFGs), when paired with the traditional parsing architecture for any given CFG, the corresponding set of accepted character-level context-free languages (CFLs) is severely restricted [3,4,5,6,7,8,9,10]. The character-level CFLs are languages defined down to the character level [4,5]. This prompted the development of the following approaches:

(1)

Scannerless architectures, which solve the problem by abandoning the use of the scanner [4,5,11].

(2)

Context-dependent scanner-based architectures, where the scanner receives the contextual information from the parser [6,7,8,9]. Most are deterministic, which means they are still dependent on the ad hoc disambiguation rules.

(3)

Nondeterministic scanner-based architectures, where the problem of resolving the non-determinism in the scanner is offloaded to a non-deterministic parser [8,12,13,14].

The contribution of this work is a novel scanner-based architecture, which is a fusion of approaches (2) and (3), which for any given CFG accepts all corresponding character-level CFLs. That means it does not require any disambiguation rules to resolve lexical conflicts. However, as our architecture captures all lexical ambiguities for certain applications, disambiguation rules are still useful to limit the number of interpretations. It can directly parse all possible specifications consisting of grammar and regular definitions. Our architecture is based on the right-nulled generalized LR (RNGLR) parser [15,16,17], and is a direct generalization of the context-aware scanning architecture by Van Wyk [6] and Schwerdfeger [6,18]. To our knowledge, this is the first practical architecture using a scanner with such power. The only other practical approach that for any given CFG accepts all corresponding character-level CFLs is the scannerless GLR (SGLR) [5,11]. However it is based on the approach in (1) and, therefore, does not use a scanner.

We share the sentiment of Tomita [15], that is, focus should be placed on the nearly deterministic languages. Virtually all programming languages fall into this category. Our architecture is geared for this use case. It degenerates to context-aware scanning for the deterministic segments, otherwise the performance degrades gracefully. For the nearly deterministic languages, it performs in predominantly linear time.

Conceptually, our architecture has an unbounded parser and scanner lookahead. This is a direct consequence of the fact that the parser and the scanner are both non-deterministic. The tables for our architecture are almost the same as the ones for the context-aware scanning architecture. The difference is that the parsing tables can have a few additional entries for $\epsilon$-shift actions. Unlike other architectures based on approach (3), our architecture is streaming; thus, the characters are processed one by one. As a result, it is also usable in online applications [15]. That is, the characters can be processed as they are typed without buffering.

The CFLs are closed under union and concatenation, and a neutral element exists for both operations. Our architecture for any given CFG accepts all character-level CFLs and, thus, preserves these properties and the related identities, and, by extension, all other operators that are based on them, such as the Kleene star. For the lexical part of the specification, we used the extended regular expressions [19,20]; that means, the complement, intersection, and difference are supported there as well.

These benefits mean that the grammar writers have more freedom since they are not constrained to the grammars where the lexical syntax can be disambiguated deterministically. The grammars are also less fragile because they do not depend on a careful selection of disambiguation rules to be parsed. That way, the modifications to the existing grammar can be performed more easily.

Our architecture can be utilized for:

• Legacy programming languages, such as COBOL and Fortran. Additionally, POSIX shell [21] could be supported as well (with a few extensions for the syntax that is not context-free). These languages were developed before the advent of modern parsing methods. Therefore, there was no inclination for the language to be parsed using deterministic methods, such as LR. However, even modern programming languages most commonly cannot be parsed using deterministic methods without ad hoc disambiguation rules and/or modifications to the grammar [11,14].
• Composite programming languages. These are programming languages with embedded sublanguages. The sublanguages usually have a different lexical syntax than the host language. The problem is solved traditionally by surrounding the sublanguage with sentinels. The scanner then recognizes that sublanguage as a single lexeme, which is then processed separately, or it switches to a different scanner mode when it encounters a sentinel. Simple omnipresent examples are the escape sequences in strings and string interpolation [7,14,22]. Further examples where sublanguages are pervasive are:

  (a)

  HTML, which allows the embedding of JavaScript and CSS.

  (b)

  PHP, which is itself embedded in HTML [7].

  (c)

  Parser and scanner specifications, which include the sublanguages for specifying actions [7].

  (d)

  Tex, where each environment can be its own sublanguage.

• Extensible languages. These are languages that allow extensions of the syntax to be specified modularly. This allows embedding of arbitrary domain-specific languages [23]. That way, even non-experts can extend the language [6,7,22,24,25,26].
• Language workbenches. These allow for the automatic generation of interactive environments and related language processing and manipulation tools for domain-specific languages [23]. Traditionally these have utilized scannerless architectures [27,28,29].
• Natural language applications. GLR parsing was originally developed for this domain [15,16,30].

All of these use cases are subsumed by supporting all possible specifications, although some require a more sophisticated metalanguage than the one we use. Such metalanguages have been studied extensively in other works [5,6,7,14,18,27,28,29].

The paper is organized as follows. In Section 2, we introduce the basic notational conventions used in this work. In Section 3, we introduce the preliminaries. In Section 3.1, we introduce RNGLR parsing. In Section 3.2, we introduce the traditional parsing architecture briefly to establish the terminology and the related notation. In Section 3.3, we introduce context-aware scanning, which presents the groundwork for our architecture. In Section 4, we present our architecture. The construction of the parse forest is discussed in Section 5. In Section 6, we present the addition of the scanner lookahead as an optimization. In Section 7, we provide a formal proof of correctness. In Section 8, we describe our experience using our architecture. Finally, Section 9 follows the discussion of the related work, and the conclusion is in Section 10.

2. Basic Notation

We use the notation introduced in the Dragon book [2] with the following modifications/additions: Lowercase letters from the end of the Greek alphabet $\upsilon ,\psi ,\omega$ are used for strings of terminal symbols. The letters $v,u,w$ are used for the graph-structured stack (GSS) vertices. The letter z is used for the shared packed parse forest (SPPF) nodes. The letters $t,l$ are additionally used for the terminal symbols. The letters $M,\dot{M},\mathring{M}$ are used for automata. Bold uppercase letters from the Latin or the Greek alphabets $\mathbf{S},\mathsf{\Psi }$ are used for sets. Fraktur lowercase letters from the Latin alphabet $\mathfrak{i},\mathfrak{j},\mathfrak{k}$ are used for the positions in the input string. Cursive uppercase letters from the Latin alphabet $\mathcal{S}$ are used for mutable references to sets. That means that the set itself cannot change while its elements are iterated; however, $\mathcal{S}$ will point to different sets during the execution of the algorithm. The $◊$ is used to denote initial elements. For example, we used $q_{◊}$ for the start state of the automaton, instead of $q_0$.

We use the following operators: $\mathcal{P}\left(\mathbf{S}\right)$ is the power set of a set. ${\mathbf{S}}^{*}$ is the Kleene closure of a set. ${\mathbf{S}}^{+}$ is the positive closure of a set. ${\mathbf{R}}^{★}$ is the transitive reflexive closure of a relation. $\left|\omega \right|$ is the length of a string. $\left|\mathbf{S}\right|$ is the size of a set. ↛ is a partial function, which means it only maps a subset of its domain. $\mathbf{S}\times \mathbf{P}$ is the cross-product between two sets. ${\omega }_{(\mathfrak{i},\mathfrak{j})}$ is a substring of $\omega$ starting at $\mathfrak{i}$ and ending at $\mathfrak{j}$. The character at $\mathfrak{i}$ is not included, and $\mathfrak{j}$ is included in the substring.

An indexed family is a collection of elements from a set $\mathbf{S}$ indexed by a set $\mathbf{I}$, denoted as ${\left(s_i\right)}_{i\in \mathbf{I}}$. It is equivalent to a function, $\iota :\mathbf{I}\to \mathbf{S}$, such that $\iota \left(i\right)=s_i$. A tuple $(s_1,\dots ,s_n)$ is a family indexed by a set $\{1,\dots ,n\}$, and can be written as ${\left(s_i\right)}_{i\in \{1,\dots ,n\}}$. Appending to a tuple will be denoted as $(s_1,\dots ,s_n)⌢s^{\prime }=(s_1,\dots ,s_n,s_{n+1}^{\prime })$.

3. Preliminaries

A CFG defined over a finite set of symbols $\mathbf{\Sigma }$ is a 4-tuple $G=(\mathbf{N},\mathbf{\Sigma },\mathbf{R},S)$, where $\mathbf{N}$ is a finite set of non-terminal symbols, $\mathbf{\Sigma }$ is a finite set of terminal symbols, such that $\mathbf{N}\cap \mathbf{\Sigma }=\varnothing$, $\mathbf{R}\in \mathcal{P}(\mathbf{N}\times {(\mathbf{N}\cup \mathbf{\Sigma })}^{*})$ is a relation that contains production rules of the form $A::=\beta$ and $S\in \mathbf{N}$ is the start symbol. A regular grammar (RG) is a CFG with the production rules restricted to $A::=\psi$ and $A::=\psi B$, where $A,B\in \mathbf{N}\wedge \psi \in {\mathbf{\Sigma }}^{*}$. Character-level CFG is a CFG over a finite set of characters $\mathbf{\Omega }=\{\mathtt{a},\mathtt{b},\mathtt{c},\dots \}$ [4,5]. A string of characters is any $\omega \in {\mathbf{\Omega }}^{*}$.

A derivation step for the production rule $A::=\beta$ is $\gamma A\eta \Rightarrow \gamma \beta \eta$, where $\gamma ,\eta \in {(\mathbf{N}\cup \mathbf{\Sigma })}^{*}$. A derivation of $\tau$ from $\sigma$ is a sequence of derivation steps $\sigma \Rightarrow \cdots \Rightarrow \tau$, denoted as $\sigma \stackrel{*}{\Rightarrow }\tau$ or $\sigma \stackrel{+}{\Rightarrow }\tau$ if the sequence is non-empty. A string $\sigma \in {(\mathbf{N}\cup \mathbf{\Sigma })}^{*}$ where $S\stackrel{*}{\Rightarrow }\sigma$ is called a sentential form. A sentential form $\omega \in {\mathbf{\Sigma }}^{*}$ is called a sentence. The first_k set of a string $\sigma \in {(\mathbf{N}\cup \mathbf{\Sigma })}^{*}$ is defined as:

$${\mathrm{FIRST}}_k(\sigma )=\{\omega \in {\mathbf{\Sigma }}^{*}\mid \sigma \stackrel{*}{\Rightarrow }\omega \psi \wedge |\omega |\le k\}$$

A CFL generated by a context-free grammar G is the set of all strings that can be produced by G and is defined as $\mathcal{L}\left(G\right)=\{\omega \in {\mathbf{\Sigma }}^{*}\mid S\stackrel{*}{\Rightarrow }\omega \}$. The CFGs have an independent production property, which means that the derivation steps do not depend on the surrounding context. Therefore, we can extend the definition of the language to any non-terminal symbol A in the grammar, $\mathcal{L}\left(A\right)=\{\omega \in {\mathbf{\Sigma }}^{*}\mid A\stackrel{*}{\Rightarrow }\omega \}$ [1]. The language of a terminal symbol $a\in \mathbf{\Sigma }$ is $\mathcal{L}\left(a\right)=\left\{a\right\}$. Regular language (RL) is a language generated by an RG.

For $\omega ,\psi \in {\mathbf{\Sigma }}^{*}$, $\psi$ is a prefix of $\omega$ if $\exists {\omega }^{\prime }\in {\mathbf{\Sigma }}^{*},\omega =\psi {\omega }^{\prime }$. If ${\omega }^{\prime }\ne \epsilon$, then $\psi$ is a proper-prefix of $\omega$. For $\omega ,\psi \in {\mathbf{\Sigma }}^{*}$, $\psi$ is a suffix of $\omega$ if $\exists {\omega }^{\prime }\in {\mathbf{\Sigma }}^{*},\omega ={\omega }^{\prime }\psi$.

A language $\mathbf{L}$ is prefix-free if, for all non-empty ${\psi }_1,{\psi }_2\in \mathbf{L}$, neither ${\psi }_1$ is a proper prefix of ${\psi }_2$ nor ${\psi }_2$ is a proper prefix of ${\psi }_1$ [31]. The languages ${\mathbf{L}}_1$ and ${\mathbf{L}}_2$ are prefix-disjoint ${\mathbf{L}}_1{\asymp }_{\mathrm{pref}}{\mathbf{L}}_2$ if for all non-empty ${\psi }_1\in {\mathbf{L}}_1$ and ${\psi }_2\in {\mathbf{L}}_2$, neither ${\psi }_1$ is a proper prefix of ${\psi }_2$ nor ${\psi }_2$ is a proper prefix of ${\psi }_1$ [32]. The languages ${\mathbf{L}}_1$ and ${\mathbf{L}}_2$ are disjoint ${\mathbf{L}}_1\asymp {\mathbf{L}}_2$, if ${\mathbf{L}}_1\cap {\mathbf{L}}_2=\varnothing$.

The (extended) regular expressions (RE) over a finite alphabet $\mathbf{\Sigma }$ are defined recursively as follows:

• ∅, $\epsilon$ and $a\in \mathbf{\Sigma }$ are RE.
• If r and s are RE, then their concatenation $\left(rs\right)$, union $\left(r\right|s)$, Kleene star ${\left(r\right)}^{*}$, Kleene plus ${\left(r\right)}^{+}$, and as an extension complement $\neg \left(r\right)$, intersection $(r\&s)$, and difference $(r-s)$ are RE.

They denote the following languages:

$$\begin{array}{lllllllll}\mathcal{L}(⌀)& =& \varnothing & \mathcal{L}\left(rs\right)& =& \{\omega \psi \mid \omega \in \mathcal{L}(r)\wedge \psi \in \mathcal{L}(s\left)\right\}& \mathcal{L}(\neg r)& =& {\mathbf{\Sigma }}^{*}-\mathcal{L}\left(r\right)\\ \mathcal{L}\left(\epsilon \right)& =& \left\{\epsilon \right\}& \mathcal{L}\left(r\right|s)& =& \mathcal{L}\left(r\right)\cup \mathcal{L}\left(s\right)& \mathcal{L}(r\&s)& =& \mathcal{L}\left(r\right)\cap \mathcal{L}\left(s\right)\\ \mathcal{L}\left(a\right)& =& \left\{a\right\}& \mathcal{L}\left(r*\right)& =& \left\{\epsilon \right\}\cup \mathcal{L}(r r*)& \mathcal{L}(r-s)& =& \mathcal{L}\left(r\right)-\mathcal{L}\left(s\right)\\ & & & \mathcal{L}\left(r^{+}\right)& =& \mathcal{L}(r r^{+})\end{array}$$

A deterministic finite automaton (DFA) is a 5-tuple $M=(\mathbf{Q},\mathbf{\Sigma },\delta ,q_{◊},\mathbf{F})$, where $\mathbf{Q}$ is a finite set of states, $\mathbf{\Sigma }$ is a finite alphabet, $\delta :\mathbf{Q}\times \mathbf{\Sigma }\to \mathbf{Q}$ is a transition function, $q_{◊}\in \mathbf{Q}$ is the start state and $\mathbf{F}\subseteq \mathbf{Q}$ is a set of final states. A DFA is complete if $\delta (q,a)$ is defined for all $q\in \mathbf{Q}$ and $a\in \mathbf{\Sigma }$. The transition function of a DFA can be extended to ${\delta }^{*}:\mathbf{Q}\times {\mathbf{\Sigma }}^{*}\nrightarrow \mathbf{Q}$:

$$\begin{array}{cc}\hfill {\delta }^{*}(q,\epsilon )& =q\hfill \\ \hfill {\delta }^{*}(q,a\omega )& ={\delta }^{*}(q^{\prime },\omega )\mathrm{when}{q}^{\prime }=\delta (q,a)\mathrm{is}\mathrm{defined}\hfill \end{array}$$

The language of a DFA is defined as $\mathcal{L}\left(M\right)=\{\omega \in {\mathbf{\Sigma }}^{*}\mid {\delta }^{*}(q_{◊},\omega )\in \mathbf{F}\}$ [33]. A DFA corresponding to an empty language will be denoted as ⌀, such that $\mathcal{L}(⌀)=\varnothing$. The path in the DFA $q_0,\dots ,q_k$, such that for all $i\in \{0,\dots ,k-1\},\exists a\in \mathbf{\Sigma },q_{i+1}=\delta (q_i,a)$, will be denoted as $q_0\to \dots \to q_k$, or as $q_0\stackrel{\psi }{\to }{q}_k$, where $q_k={\delta }^{*}(q_0,\psi )$. For the following constructions, the DFAs need to be complete.

The union DFA of an indexed family of DFA is $M_1,\dots ,M_n$, $M_1|\dots |M_n=({\mathbf{Q}}_1\times \cdots \times {\mathbf{Q}}_n,\mathbf{\Sigma },\delta ,(q_{◊,1},\dots ,q_{◊,n}),\mathbf{F})$, where $\delta ((q_1,\dots ,q_n),a)=({\delta }_1(q_1,a),\dots ,{\delta }_n(q_n,a))$ and $\mathbf{F}=\{(q_1,\dots ,q_n)\mid q_1\in {\mathbf{F}}_1\vee \dots \vee q_n\in {\mathbf{F}}_n\}$. The reflexive transitive closure of $\delta$, ${\delta }^{★}:\mathbf{Q}\to \mathcal{P}\left(\mathbf{Q}\right)$, is the least solution of the following set equation:

$${\delta }^{★}\left(q\right)=\left\{q\right\}\cup \bigcup _{a\in \mathbf{\Sigma }}{\delta }^{★}\left(\delta (q,a)\right)$$

A set of dead states ${\mathbf{Q}}_{†}$ contains non-final states from which no final state can be reached, ${\mathbf{Q}}_{†}=\{q\in \mathbf{Q}\mid {\delta }^{★}\left(q\right)\cap \mathbf{F}=\varnothing \}$. The $\mathrm{FIRST}$ set of an automaton M, such that $\mathrm{FIRST}\left(M\right)=\mathrm{FIRST}\left(\mathcal{L}\right(M\left)\right)$ can be defined as $\mathrm{FIRST}\left(M\right)=\{a\mid a\in \mathbf{\Sigma }\wedge \delta (q_{◊},a)\notin {\mathbf{Q}}_{†}\}$.

A complete DFA with a variable start state is defined as $M\langle q\rangle =(\mathbf{Q},\mathbf{\Sigma },\delta ,q\in \mathbf{Q},\mathbf{F})$. Recognizing $\psi$ using any $M\langle q\rangle$, ${\delta }^{*}(q,\psi )\in \mathbf{F}$, can be split into two steps. Let $\psi ={\psi }_1{\psi }_2$, where ${\psi }_1,{\psi }_2\in {\mathbf{\Sigma }}^{*}$, first, transition to state $q^{\prime }={\delta }^{*}(q,{\psi }_1)$ using $M\langle q\rangle$, and second, recognize the suffix ${\psi }_2$ using $M\langle q^{\prime }\rangle$, ${\delta }^{*}(q^{\prime },{\psi }_2)\in \mathbf{F}$. Notice that the second step can be split again. If we iterate this process k times, we can recognize $\psi ={\psi }_1\dots {\psi }_k$, where ${\psi }_1,\dots ,{\psi }_k\in {\mathbf{\Sigma }}^{*}$, in k steps.

For each RL, there exists an RG G, a RE r, and a DFA M, such that $\mathcal{L}\left(G\right)=\mathcal{L}\left(r\right)=\mathcal{L}\left(M\right)$. That is, all of the representations have the same expressive power, and can be converted into one another [1,2].

3.1. RNGLR Parsing

RNGLR [17] parsing for a grammar G over $\mathbf{\Sigma }$ is an extension of GLR parsing [15,16], which in turn is a generalization of LR parsing, which can parse all CFGs while remaining efficient on (near)-deterministic ones. It uses the handle-finding automaton for LR parsing extended with short circuiting-reduce actions for the right-nulled productions of the form $A::=\gamma \eta$, where $\eta \stackrel{*}{\Rightarrow }\epsilon$. Refer to [17] for the discussion on why that is required. The handle-finding automaton is defined as $(\mathbf{S},\mathbf{N}\cup \mathbf{\Sigma },{\mathcal{T}}_G,s_{◊},\mathbf{H})$. The transition function is called the goto table ${\mathcal{T}}_G:\mathbf{S}\times (\mathbf{N}\cup \mathbf{\Sigma })\nrightarrow \mathbf{S}$. Each state $s\in \mathbf{S}$ of the automaton is labeled with a set of items. An item $(A::=\gamma •\eta a)$ is a production with a dot in the right-hand-side, which delimits the symbols that have already been seen from those that have not. The $a\in {\mathbf{\Sigma }}_{\$}$ is the lookahead symbol, where ${\mathbf{\Sigma }}_{\$}=\mathbf{\Sigma }\cup \left\{\$\right\}$ is the set of all lookahead symbols (the $ is the end-of-input marker). The sets of items are used to construct the action table ${\mathcal{T}}_A:\mathbf{S}\times {\mathbf{\Sigma }}_{\$}\to \mathcal{P}(\mathbf{\Delta })$, where $\mathbf{\Delta }$ is the set of all actions. For each state $s\in \mathbf{S}$ the shift items $(A::=\gamma •a{\eta }^{\prime }a)\in s$ are converted into $\left\{\right(s,a)\mapsto \mathrm{SHIFT}\}$, and the reduce items $(A::=\gamma •\eta a)\in s$, where $\eta \stackrel{*}{\Rightarrow }\epsilon$ are converted into $\left\{\right(s,a)\mapsto \mathrm{REDUCE}(A,\left|\gamma \right|\left)\right\}$. The grammar needs to be augmented with the production rule $S^{\prime }::=S$. It results in items $(S^{\prime }::=S•\$)\in s$ and possibly $(S^{\prime }::=•\$)\in s$, which are converted into $\left\{\right(s,\$)\mapsto \mathrm{ACCEPT}\}$. The state whose label contains it is the final state $s\in \mathbf{H}$ of the automaton [1,2]. To construct the automaton, multiple methods exist, such as: LR, SLR, LALR [1], and IELR [7]. In this paper, LALR is used; however, the algorithms discussed apply to all of them.

The handle-finding automaton by itself is deterministic; however, more than one action may be possible in each state. If more than one reduce action is possible, there is a reduce/reduce conflict, and if a shift and a reduce action are possible, there is a shift/reduce conflict. We call these states inadequate. The presence of such states results in non-determinism, which is resolved in RNGLR parsing using a breadth-first search. Conceptually, when there are multiple actions possible in a given state the process splits, and each action is performed in its own branch on its own copy of the stack. Thus, the algorithm performs all possible traversals of the handle-finding automaton. The processes are synchronized, they perform all reduce actions at the given position, shift the next input symbol, and then all move to the next position at the same time. If there are no inadequate states, the process remains deterministic; in that case, the algorithm preserves the linear worst-case complexity of LR parsing [1,15,16].

Copying the stack would result in an exponential blow-up; thus, instead, a directional acyclic graph is used to keep track of the states, called the graph-structured stack (GSS) $\Gamma =(\mathbf{V},\mathbf{E})$. It allows combining the common parts of the stacks, and as a result, prevents repeating the same work multiple times. The vertices $\mathbf{V}$ are labeled with the automaton states $\mathbf{S}$. The function $\mathrm{STATE}:\mathbf{V}\to \mathbf{S}$ retrieves the label $s\in \mathbf{S}$ corresponding to the vertex $v\in \mathbf{V}$. For a subset of vertices, $\mathbf{W}\subseteq \mathbf{V}$ the function $\mathrm{STATE}:\mathcal{P}\left(\mathbf{V}\right)\to \mathcal{P}\left(\mathbf{S}\right)$ is defined as $\mathrm{STATE}\left(\mathbf{W}\right)=\left\{\mathrm{STATE}\right(w)\mid w\in \mathbf{W}\}$. The edges $\mathbf{E}\subseteq \mathbf{V}\times \mathbf{V}$ are unlabeled. We denote edges $(w,v)\in \mathbf{E}$ as $v\leftarrow w$. In the diagrams, each edge $(w,v)$ is labeled redundantly with the symbol that was used to make the transition $X\in (\mathbf{N}\cup \mathbf{\Sigma })$, where $\mathrm{STATE}\left(w\right)={\mathcal{T}}_G(\mathrm{STATE}\left(v\right),X)$, to make the presentation clearer. The graph paths $w_1,\dots ,w_k$, where, for all $i\in \{1,\dots ,k-1\},(w_i,w_{i+1})\in \mathbf{E}$, are denoted as $w_k\stackrel{\alpha }{\leftarrow }{w}_1$, where $\mathrm{STATE}\left(w_1\right)={\mathcal{T}}_G^{*}(\mathrm{STATE}\left(w_k\right),\alpha )$.

The vertices are partitioned into subclasses ${\left({\mathbf{U}}_{\mathfrak{i}}\right)}_{\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}}$. Each ${\mathbf{U}}_{\mathfrak{i}}\subseteq \mathbf{V}$ is a set of vertices created when parsing $a_{\mathfrak{i}}$, where $\mathfrak{i}$ represents the position in the input string $\omega \in {\mathbf{\Sigma }}^{*}$. The special case is ${\mathbf{U}}_0$, which is created at the start, before processing $\omega$. It contains a vertex $v_{◊}$, labeled with the start state of the handle-finding automaton $s_{◊}$, called the bottom. The states that label paths $v_{◊}\stackrel{\alpha }{\leftarrow }w$ in $\Gamma$ correspond directly to stacks $\alpha$ in LR parsing. The tops of the stacks at position $\mathfrak{i}$ are states in $\mathrm{STATE}\left({\mathbf{U}}_{\mathfrak{i}}\right)$.

The algorithm works by traversing $\omega \$$, where $\omega =a_1\dots a_{\left|\omega \right|}$, from left to right. At each position $\mathfrak{i}$, there is a lookahead symbol $a_{\mathfrak{j}}$, where $\mathfrak{j}$ represents the next position, $\mathfrak{j}=\mathfrak{i}+1$. First, the vertices $w\in {\mathbf{U}}_{\mathfrak{i}}$, such that $\mathrm{REDUCE}(A,|\gamma \left|\right)\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),a_{\mathfrak{j}})$ are identified. The reduction step is performed by finding all paths $v\stackrel{\gamma }{\leftarrow }w$ in $\Gamma$ of length $\left|\gamma \right|$. Then for a vertex $v\in {\mathbf{U}}_{\mathfrak{h}\in \{0,\dots ,\mathfrak{i}\}}$ a vertex $u\in {\mathbf{U}}_{\mathfrak{i}}$ labeled with ${\mathcal{T}}_G(\mathrm{STATE}\left(v\right),A)$ is either found or created, and the vertices are connected with an edge $v\leftarrow u$ (if one does not already exist). In the case of $\epsilon$-reduce actions, only a single path can be found $v=w$, since $\left|\gamma \right|=0$. For the vertex, the $w\in {\mathbf{U}}_{\mathfrak{i}}$ a vertex $u\in {\mathbf{U}}_{\mathfrak{i}}$ is either found or created, and the vertices are connected with an edge $w\leftarrow u$. Since there can be no more than one such edge, the $\epsilon$-reduce actions only need to be considered when a new vertex is created. When a new vertex $u\in {\mathbf{U}}_{\mathfrak{i}}$ is created as a result of $\epsilon$-reduce, the reduce actions where $\left|\gamma \right|>0$ must not be considered since they are already applied from one of its descendants $w^{\prime }\in {\mathbf{U}}_{\mathfrak{i}}$ [17]. The process is repeated as new vertices and/or edges are added, until there are no possible reduce actions left. Then, vertices $w\in {\mathbf{U}}_{\mathfrak{i}}$ such that $\mathrm{SHIFT}\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),a_{\mathfrak{j}})$ are identified. For each vertex w, a vertex $u\in {\mathbf{U}}_{\mathfrak{j}}$ labeled with ${\mathcal{T}}_G(\mathrm{STATE}\left(w\right),a_{\mathfrak{j}})$ is either found or created, and the vertices are connected with an edge $w\leftarrow u$ (if one does not already exist). Then the process is repeated for ${\mathbf{U}}_{\mathfrak{j}}$. If there exists $w\in {\mathbf{U}}_{\left|\omega \right|}$, such that $\mathrm{ACCEPT}\in {\mathcal{T}}_A$$\left(\mathrm{STATE}\right(w),\$)$, the parsing completed successfully [15,16].

3.2. Traditional Parsing Architecture

Traditional parsing architecture for parsing character strings $\omega \in {\mathbf{\Omega }}^{*}$ is a two-phase process. The first phase, which is based on character-level RE, is called the lexical analysis and is performed by the scanner. The second phase, which is based on CFG, is called the syntactic analysis, and is performed by the parser. For directional parsers the processes of both phases can be interleaved, which is the case we will focus on.

The parser and the scanner are traditionally constructed using a specification, defined as a pair:

$$\Xi =(G^{\prime },{\left(r_t\right)}_{t\in \mathbf{T}})$$

It is composed of a CFG $G^{\prime }=({\mathbf{N}}^{\prime },\mathbf{T},{\mathbf{R}}^{\prime },S)$ over a finite set of symbols $\mathbf{T}$ and character-level RE ${\left(r_t\right)}_{t\in \mathbf{T}}$ over characters $\mathbf{\Omega }$ for each terminal symbol $t\in \mathbf{T}$ in $G^{\prime }$, which are called regular definitions [2]. A derivation step for $A\in {\mathbf{N}}^{\prime }$ is $\gamma A\eta \Rrightarrow \gamma \beta \eta$, where $\gamma ,\eta \in {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}$ and there exists a production rule $A::=\beta$, and a derivation step for $t\in \mathbf{T}$ is $\sigma t\kappa \Rrightarrow \sigma \psi \kappa$, where $\sigma ,\kappa \in {(\mathbf{T}\cup \mathbf{\Omega })}^{*}$, and $\psi \in \mathcal{L}\left(r_t\right)$. The language generated by $\Xi$ is defined as $\mathcal{L}(\Xi )=\{\omega \in {\mathbf{\Omega }}^{*}\mid S\stackrel{*}{\Rrightarrow }\omega \}$. It is easy to see $\Xi$ generates a character-level CFL. For each $\Xi$, there exists a corresponding character-level CFG $\widehat{G}=({\mathbf{N}}^{\prime }\cup \mathbf{T}\cup \mathbf{Y},\mathbf{\Omega },{\mathbf{R}}^{″},S)$, such that $\mathcal{L}\left(\widehat{G}\right)=\mathcal{L}(\Xi )$. There is a production rule $A::=\beta \in {\mathbf{R}}^{″}$ for each $A::=\beta \in {\mathbf{R}}^{\prime }$. The RE ${\left(r_t\right)}_{t\in \mathbf{T}}$ are converted into RG and the resultant productions are added to ${\mathbf{R}}^{\prime }$. The non-terminal symbols created during the conversion correspond to $\mathbf{Y}$. Due to the independent production property, a conversion in the reverse direction is possible as well.

The scanner automaton is constructed by converting ${\left(r_t\right)}_{t\in \mathbf{T}}$ into DFA ${\left(M_t\right)}_{t\in \mathbf{T}}$, where $M_t=({\mathbf{Q}}_t,\mathbf{\Omega },{\delta }_t,q_{◊,t},{\mathbf{F}}_t)$, such that $\mathcal{L}\left(M_t\right)=\mathcal{L}\left(r_t\right)$. To preserve the association between the terminal symbols and the DFA a function $m:{\mathbf{Q}}_t\to \mathcal{P}\left(\mathbf{T}\right)$ is introduced:

$$m\left(q_t\right)=\left\{\begin{array}{cc}\left\{t\right\}\hfill & \mathrm{if}{q}_t\in {\mathbf{F}}_t\hfill \\ \varnothing \hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

It maps a state $q_t\in {\mathbf{Q}}_t$ to the corresponding terminal symbol $t\in \mathbf{T}$, if $q_t\in {\mathbf{F}}_t$. The scanner uses an automaton ${\dot{M}}_{\mathbf{T}}={|}_{t\in \mathbf{T}}{M}_t$, where:

$${\dot{M}}_{\mathbf{T}}=(\dot{\mathbf{Q}}=\underset{t\in \mathbf{T}}{\times }{\mathbf{Q}}_t,\mathbf{\Omega },\dot{\delta },{\dot{q}}_{◊}={\left(q_{◊,t}\right)}_{t\in \mathbf{T}},\dot{\mathbf{F}})$$

${\dot{M}}_{\mathbf{T}}$ is renumbered for execution, thus, the connection between each $\dot{q}\in \dot{\mathbf{Q}}$ and the corresponding ${\left(q_t\right)}_{t\in \mathbf{T}}$ is not preserved. Therefore, the states are labeled with $m\left(\dot{q}\right)={\bigcup }_{t\in \mathbf{T}}m\left(q_t\right)$, as they can no longer be determined after renumbering.

To combine RNGLR parsing with the traditional parsing architecture, the scanner automaton is constructed for ${\mathbf{T}}_{\$}$, where $r_{\$}=\$$. The $ is a character not appearing in $\mathbf{\Omega }$. To parse the input string $\omega \in \mathbf{\Omega }$, the algorithm works by traversing $\omega \$$, from left to right. At each position $\mathfrak{i}$, the scanner recognizes a lexeme ${\upsilon }_{(\mathfrak{i},\mathfrak{j})}\in {\mathbf{\Omega }}_{\$}^{+}$ in the rest of the input string ${\omega }^{\prime }\in {\mathbf{\Omega }}_{\$}^{+}$, such that ${\omega }^{\prime }={\upsilon }_{(\mathfrak{i},\mathfrak{j})}{\omega }^{″}$ and ${\dot{q}}_{◊}\stackrel{{\upsilon }_{(\mathfrak{i},\mathfrak{j})}}{\to }{\dot{q}}^{\prime }\in \dot{\mathbf{F}}$. The next position is $\mathfrak{j}=\mathfrak{i}+|{\upsilon }_{(\mathfrak{i},\mathfrak{j})}|$ and ${\omega }^{″}$, thus starting at position $\mathfrak{j}$. The $m\left({\dot{q}}^{\prime }\right)$ are the lookahead symbols that are matched.

• If $m\left({\dot{q}}^{\prime }\right)=\varnothing$ no lexeme is recognized and an error is raised.
• If $m\left({\dot{q}}^{\prime }\right)=\left\{t_{\mathfrak{j}}\right\}$, the lookahead symbol $t_{\mathfrak{j}}$ is passed to the parser.
• If $\left|m\right({\dot{q}}^{\prime }\left)\right|>1$, then multiple $\left(M_t\right)$ match the same lexeme and an error is raised.

The process is repeated for $\mathfrak{j}$ and ${\omega }^{″}$. Note that for all $\mathfrak{j}\in \{1,\dots ,|\omega \left|\right\}$, ${\mathbf{U}}_{\mathfrak{j}}$ are non-empty if and only if $t_{\mathfrak{j}}$ is matched and a shift action is performed for $t_{\mathfrak{j}}$. The ${\mathbf{U}}_0$ cannot be empty as it contains at least $v_{◊}$.

During scanning, two types of lexical conflicts can occur [9]:

(I)

Multiple automata can match a single lexeme. This can occur if the languages of the automata are not pairwise disjoint, $t,l\in \mathbf{T}\wedge t\ne l\wedge \mathcal{L}\left(M_t\right)\asymp \mathcal{L}\left(M_l\right)$. This type of lexical conflict corresponds to a reduce/reduce conflict in LR parsing for $\widehat{G}$ [8].

(II)

Multiple lexemes of different lengths can be recognized [7,8,18]. This can occur if the automata are not prefix-free or pairwise prefix-disjoint, $t,l\in \mathbf{T}\wedge t\ne l\wedge \mathcal{L}\left(M_t\right){\asymp }_{\mathrm{pref}}\mathcal{L}\left(M_l\right)$. This type of lexical conflict corresponds to a shift/reduce conflict in LR parsing for $\widehat{G}$ [7,8].

To keep the process deterministic, the automata ${\left(M_t\right)}_{t\in \mathbf{T}}$ must be pairwise disjoint, prefix-free, and pairwise-prefix disjoint. Since this condition is too restrictive, instead, in circumstances where the scanner is faced with multiple possible choices, one is chosen based on predefined disambiguation rules in the case of deterministic methods [6,7,8,18]. These restrictions are not included in the original definition of $\Xi$; therefore, not all $\Xi$ are accepted by the traditional parsing architecture. In this paper, we present an architecture that accepts all $\Xi$ directly and, as a result, for any given $G^{\prime }$, accepts all corresponding character-level CFL, instead of parsing $\widehat{G}$, as is the case with character-level parsers [4,5].

Let us demonstrate the combination with the following specification $\Xi$:

$$S::=Ac|bcA::=b{r}_b=\mathtt{x}\mathtt{y}{r}_c=\mathtt{z}{r}_{\$}=\$$$

(1)

The handle-finding automaton, constructed using $G^{\prime }$ in (1), is presented in Figure 1. The handle-finding automaton state $s\in \mathbf{S}$ is presented next to each vertex in the diagram and each vertex is labeled with the corresponding set of items. The scanner automaton is presented in Figure 2. Each vertex in the diagram is labeled by a scanner automaton state $\dot{q}\in \dot{\mathbf{Q}}$. Above each vertex in the diagram is the lookahead symbol that is matched $m\left(\dot{q}\right)$. The edges are labeled with sets of characters, a single character denotes a singleton set.

Parsing the input string $\omega =\mathtt{x}\mathtt{y}\mathtt{z}$ results in the GSS and the scanner trace presented in Figure 3 and Figure 4, respectively. The scanner trace outlines the paths through the scanner automaton for each recognized lexeme. The number above each start state is the position $\mathfrak{i}$ at which the automaton starts, and corresponds to ${\mathbf{U}}_{\mathfrak{i}}$ from which the shift action will be performed. The terminal symbol above each final state is the lookahead symbol that is passed to the parser. When the automaton enters a dead state, it can no longer match anything; thus, the paths end at that point. Additionally, note that there is no ${\mathbf{U}}_1$, as there are no matches at position 1.

3.3. Context-Aware Scanning Architecture

Context-aware scanning [6,18] is an extension of the traditional parsing architecture. Likewise, the scanner automaton is constructed for the lookahead symbols ${\mathbf{T}}_{\$}$. The difference is that the scanner is provided with the contextual information it lacks from the parser. The information is passed in the form of a valid lookahead set, which is a set of all lookahead symbols that are valid at a given point in the parse. The contextual information is summarized in the state of the handle-finding automaton; thus, for states $s\in \mathbf{S}$, such a set can be constructed by collecting the lookahead symbols that have actions associated with them:

$$\nu \left(s\right)=\{t\in {\mathbf{T}}_{\$}\mid {\mathcal{T}}_A(s,t)\ne \varnothing \}$$

This limits the selection of automata that need to be considered at a given point; thus, for all $s\in \mathbf{S}$, only a subset should be pairwise disjoint and pairwise prefix-disjoint $t,l\in \nu \left(s\right)\wedge t\ne l\wedge \mathcal{L}\left(M_t\right)\asymp \mathcal{L}\left(M_l\right)\wedge \mathcal{L}\left(M_t\right){\asymp }_{\mathrm{pref}}\mathcal{L}\left(M_l\right)$. As a consequence, this allows more specifications $\Xi$ to be parsed deterministically [6,18].

Let us define a function $p:{\mathbf{Q}}_t\to \mathcal{P}\left({\mathbf{T}}_{\$}\right)$ for $M_t$ as a reflexive transitive closure of m:

$$p\left(q_t\right)=\bigcup _{q_t^{\prime }\in {\delta }^{★}\left(q_t\right)}m\left(q_t^{\prime }\right)$$

It returns the lookahead symbols that could possibly be matched by continuing from $q_t\in {\mathbf{Q}}_t$. The function can easily be defined for ${\dot{M}}_{{\mathbf{T}}_{\$}}$ as $p\left(\dot{q}\right)={\bigcup }_{t\in {\mathbf{T}}_{\$}}p\left(q_t\right)$. Note that, for a path ${\dot{q}}_0\to \dots \to {\dot{q}}_k$, where ${\dot{q}}_0={\dot{q}}_{◊}$, in the automaton, the following holds $p\left({\dot{q}}_0\right)\supseteq \cdots \supseteq p\left({\dot{q}}_k\right)$. In other words, the possibilities narrow as more of the input string is seen. As the automaton enters a dead state, ${\dot{q}}^{\prime }\in {\dot{\mathbf{Q}}}_{†}$, $p\left({\dot{q}}^{\prime }\right)=\varnothing$.

Conceptually, a context-aware scanner uses a distinct scanner automaton ${\dot{M}}_s$ for each state in the handle-finding automaton $s\in \mathbf{S}$, specialized for recognizing lookahead symbols in $\nu \left(s\right)$. There exist many different ways to construct such a scanner [6,7,8,9,18]. We use an approach proposed by Van Wyk [6] and Schwerdfeger [6,18].

Instead of constructing an automaton for each handle-finding automaton state, the same automaton is used as in the traditional parsing architecture, ${\dot{M}}_{{\mathbf{T}}_{\$}}$, and the scanning process is modified in a way that only lookahead symbols from $\nu \left(s\right)$ can be recognized. The $m:\mathbf{S}\times \dot{\mathbf{Q}}\to \mathcal{P}\left({\mathbf{T}}_{\$}\right)$ and $p:\mathbf{S}\times \dot{\mathbf{Q}}\to \mathcal{P}\left({\mathbf{T}}_{\$}\right)$ are defined as:

$$\begin{array}{cc}\hfill m(s,\dot{q})& =\nu \left(s\right)\cap m\left(\dot{q}\right)\hfill \\ \hfill p(s,\dot{q})& =\nu \left(s\right)\cap p\left(\dot{q}\right)\hfill \end{array}$$

In a traditional scanner, we know that an automaton can no longer match anything when it enters a dead state. Here, this is no longer the case, since we are interested in recognizing only a subset of possible lookahead symbols. Instead, we can use the alternative definition: For some $s\in \mathbf{S}$, no more matches can be made from the state $\dot{q}\in \dot{\mathbf{Q}}$ when $p(s,\dot{q})=\varnothing$. In other words, no lookahead symbols from $\nu \left(s\right)$ could possibly be matched by continuing from $\dot{q}$. At that point, the automaton can be stopped. The automaton finds a match when ${\dot{q}}^{\prime }\in \dot{\mathbf{F}}$, then $m(s,{\dot{q}}^{\prime })$ is passed to the parser.

Note that, since possibilities narrow as more of the input string is seen, for a path ${\dot{q}}_0\to \dots \to {\dot{q}}_k$, where ${\dot{q}}_0={\dot{q}}_{◊}$ we can use the following recursive definition as an optimization:

$$\begin{array}{cc}\hfill p(s,{\dot{q}}_0)& =\nu \left(s\right)\cap p\left({\dot{q}}_0\right)\hfill \\ \hfill p(s,{\dot{q}}_j)& =p(s,{\dot{q}}_{j-1})\cap p\left({\dot{q}}_j\right)\hfill \end{array}$$

Since $m(s,{\dot{q}}^{\prime })\subseteq p\left({\dot{q}}^{\prime }\right)$, we can instead use:

$$m(s,{\dot{q}}^{\prime })=p(s,{\dot{q}}^{\prime })\cap m\left({\dot{q}}^{\prime }\right)$$

This way, the sets contain less lookahead symbols and the intersection is more efficient to compute [6,18].

The worst-case time complexity is $O(log|{\mathbf{T}}_{\$}\left|·\right|\omega \left|\right)$, due to the fact that the intersection needs to be computed at each step. For any $s\in \mathbf{S}$, initially $|p(s,{\dot{q}}_s)|=|{\mathbf{T}}_{\$}|$; however, the size of the sets decreases as possibilities narrow. The worst-case space complexity remains the same as for the traditional parsing architecture, $O\left(\right|{\mathbf{T}}_{\$}\left|·\right|{\mathbf{Q}}_{\widehat{t}}{|}^{|{\mathbf{T}}_{\$}|})$, where $|{\mathbf{Q}}_{\widehat{t}}|$ is the number of states in the largest automaton; however, the size of the automata is increased by a constant factor due to the additional label on the states for p [18].

Let us combine RNGLR parsing and context-aware scanning. The combination was already suggested by Van Wyk [6] and Schwerdfeger [6,18], and is used by the Tree-sitter [34]. In RNGLR parsing, at each point in the parse, the handle-finding automaton may be in multiple states at once. The set of all possible combinations of these states $\Pi \subseteq \mathcal{P}\left(\mathbf{S}\right)$ for a given grammar $G^{\prime }$ can be defined as:

$$\Pi =\left\{{\bigcup }_{\alpha \in \mathbf{A}}{\mathcal{T}}_G^{*}(s_{◊},\alpha )|\psi \in {\mathbf{T}}^{*}\wedge \mathbf{A}=\{\alpha \mid \alpha \in {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}\wedge S\stackrel{*}{\Rightarrow }\alpha {\omega }^{\prime }\stackrel{*}{\Rightarrow }\psi {\omega }^{\prime }\}\right\}$$

The valid lookahead set needs to be extended to each $\mathbf{P}\in \Pi$:

$$\nu \left(\mathbf{P}\right)=\bigcup _{s\in \mathbf{P}}\nu \left(s\right)$$

Conceptually, the context-aware scanner uses a distinct scanner automaton ${\dot{M}}_{\mathbf{P}}$ for each possible combination of handle-finding automaton states $\mathbf{P}\in \Pi$, specialized for recognizing lookahead symbols in $\nu \left(\mathbf{P}\right)$.

The same idea is utilized, except that now $\nu \left(\mathbf{P}\right)$ is used instead. The functions $m:\Pi \times \dot{\mathbf{Q}}\to \mathcal{P}\left({\mathbf{T}}_{\$}\right)$ and $p:\Pi \times \dot{\mathbf{Q}}\to \mathcal{P}\left({\mathbf{T}}_{\$}\right)$ are defined as:

$$\begin{array}{cc}\hfill m(\mathbf{P},\dot{q})& =\nu \left(\mathbf{P}\right)\cap m\left(\dot{q}\right)\hfill \\ \hfill p(\mathbf{P},\dot{q})& =\nu \left(\mathbf{P}\right)\cap p\left(\dot{q}\right)\hfill \end{array}$$

The worst-case time and space complexity remain unchanged.

There are no lexical conflicts if for all $\mathbf{P}\in \Pi$ the automata corresponding to the lookahead symbols in $\nu \left(\mathbf{P}\right)$ are pairwise disjoint and pairwise prefix-disjoint; that is $t,l\in \nu \left(\mathbf{P}\right)\wedge t\ne l\wedge \mathcal{L}\left(M_t\right)\asymp \mathcal{L}\left(M_l\right)\wedge \mathcal{L}\left(M_t\right){\asymp }_{\mathrm{pref}}\mathcal{L}\left(M_l\right)$.

Let us define subsets ${\left({\mathbf{U}}_{\mathfrak{i},i}\right)}_{i\in \{0,\dots ,n\}}$ of each subclass ${\mathbf{U}}_{\mathfrak{i}}$, where each ${\mathbf{U}}_{\mathfrak{i},0}$ represents the vertices created as the shift actions were processed at the previous position, and each of the following ${\left({\mathbf{U}}_{\mathfrak{i},i}\right)}_{i\in \{1,\cdots ,n\}}$ represents the vertices created after i reduce actions were processed, and n is the number of reduce actions processed in total. The corresponding sets of the handle-finding automaton states will be denoted as ${\left({\mathbf{P}}_{\mathfrak{i},i}\right)}_{i\in \{0,\dots ,n\}}$, where ${\mathbf{P}}_{\mathfrak{i},i}\in \Pi$ and ${\mathbf{P}}_{\mathfrak{i},i}=\mathrm{STATE}\left({\mathbf{U}}_{\mathfrak{i},i}\right)$. As identified by Van Wyk [6] and Schwerdfeger [6,18], the valid lookahead set for each state contains all lookahead symbols that can possibly follow at that position. That means that vertices created as a result of processing reduce actions cannot introduce any additional lookahead symbols; that is, $\nu \left({\mathbf{P}}_{\mathfrak{i},0}\right)=\cdots =\nu \left({\mathbf{P}}_{\mathfrak{i},n}\right)$. As a result, $\nu \left({\mathbf{P}}_{\mathfrak{i},0}\right)=\nu \left({\mathbf{P}}_{\mathfrak{i}}\right)$, where ${\mathbf{P}}_{\mathfrak{i}}\in \Pi$ and ${\mathbf{P}}_{\mathfrak{i}}=\mathrm{STATE}\left({\mathbf{U}}_{\mathfrak{i}}\right)$. This allows us to construct the automaton ${\dot{M}}_{{\mathbf{P}}_{\mathfrak{i},0}}$ and perform the scanning just once at each position in the parse $\mathfrak{i}$. For this reason, we introduce automata used at each position in the parse ${\left({\dot{M}}_{\mathfrak{i}}\right)}_{\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}}$, where:

$${\dot{M}}_{\mathfrak{i}}={\dot{M}}_{{\mathbf{P}}_{\mathfrak{i},0}}$$

The scanning process remains the same as in the traditional parsing architecture, only the scanner automaton is modified. As a demonstration, let us use the following specification $\Xi$ as an example:

$$\begin{array}{cc}\hfill S& ::=Aed\left|Bef\right|ceg\hfill \\ \hfill A& ::=c\hfill \\ \hfill B& ::=c\hfill \end{array}\begin{array}{cc}\hfill r_c& =\mathtt{x}\hfill \\ \hfill r_d& =\mathtt{y}\hfill \\ \hfill r_e& =\mathtt{x}\hfill \end{array}\begin{array}{cc}\hfill r_f& =\mathtt{z}\hfill \\ \hfill r_g& =\mathtt{x}\mathtt{x}\hfill \\ \hfill r_{\$}& =\$\hfill \end{array}$$

(2)

The handle-finding automaton, constructed using $G^{\prime }$ in (2), is presented in Figure 5, and the scanner automaton is presented in Figure 6. Recognizing the input string $\omega =\mathtt{x}\mathtt{x}\mathtt{y}$ results in the GSS and the scanner trace in Figure 7 and Figure 8, respectively.

The valid lookahead sets for each handle-finding automaton state are $\nu \left(0\right)=\left\{c\right\}$ and $\nu \left(1\right)=\nu \left(4\right)=\nu \left(7\right)=\nu \left(10\right)=\left\{\$\right\}$ and $\nu \left(2\right)=\nu \left(5\right)=\nu \left(8\right)=\left\{e\right\}$ and $\nu \left(3\right)=\left\{d\right\}$ and $\nu \left(6\right)=\left\{f\right\}$ and $\nu \left(9\right)=\left\{g\right\}$. They are used to determine the valid lookahead sets $\nu \left({\mathbf{P}}_{\mathfrak{i},0}\right)$ at each position $\mathfrak{i}$ based on the sets of states ${\mathbf{P}}_{\mathfrak{i},0}$, both of which are given in 

Table 1. The sets of states ${\mathbf{P}}_{\mathfrak{i},0}$ and the valid lookahead sets $\nu \left({\mathbf{P}}_{\mathfrak{i},0}\right)$ for the specification $\Xi$ in (2).

$\mathfrak{i}$	$P_{\mathfrak{i},0}$	$\mathit{\nu }\left(P_{\mathfrak{i},0}\right)$
0	0	c
1	8	e
2	$3,6,9$	$d,f,g$
3	4	$

.

   Each row in the scanner trace represents the path traced by the automaton ${\dot{M}}_{\mathfrak{i}}={\dot{M}}_{{\mathbf{P}}_{\mathfrak{i},0}}$. At position 0, the scanning is performed using the automaton ${\dot{M}}_0={\dot{M}}_{\left\{0\right\}}$. The automaton starts in state 0, then a transition $0\stackrel{\mathtt{x}}{\to }1$ is performed.

$$\begin{array}{cc}\hfill m\left(\right\{0\},1)& =\nu \left(\right\{0\left\}\right)\cap m\left(1\right)=\left\{c\right\}\cap \{c,e\}=\left\{c\right\}\hfill \\ \hfill p\left(\right\{0\},1)& =\nu \left(\right\{0\left\}\right)\cap p\left(1\right)=\left\{c\right\}\cap \{c,e\}=\left\{c\right\}\hfill \end{array}$$

Then a transition $1\stackrel{\mathtt{x}}{\to }5$ is performed.

$$\begin{array}{cc}\hfill m\left(\right\{0\},5)& =p\left(1\right)\cap m\left(5\right)=\left\{c\right\}\cap \varnothing =\varnothing \hfill \\ \hfill p\left(\right\{0\},5)& =p\left(1\right)\cap p\left(5\right)=\left\{c\right\}\cap \varnothing =\varnothing \hfill \end{array}$$

At position 1, the scanning is performed using the automaton ${\dot{M}}_1={\dot{M}}_{\left\{8\right\}}$.

The automaton starts in state 0, then a transition $0\stackrel{\mathtt{x}}{\to }1$ is performed.

$$\begin{array}{cc}\hfill m\left(\right\{8\},1)& =\nu \left(\right\{8\left\}\right)\cap m\left(1\right)=\left\{e\right\}\cap \{c,e\}=\left\{e\right\}\hfill \\ \hfill p\left(\right\{8\},1)& =\nu \left(\right\{8\left\}\right)\cap p\left(1\right)=\left\{e\right\}\cap \{c,e\}=\left\{e\right\}\hfill \end{array}$$

Then a transition $1\stackrel{\mathtt{y}}{\to }6$ is performed.

$$\begin{array}{cc}\hfill m\left(\right\{8\},6)& =p\left(\right\{8\},1)\cap m\left(6\right)=\left\{c\right\}\cap \varnothing =\varnothing \hfill \\ \hfill p\left(\right\{8\},6)& =p\left(\right\{8\},1)\cap p\left(6\right)=\left\{c\right\}\cap \varnothing =\varnothing \hfill \end{array}$$

At position 2, the scanning is performed using the automaton ${\dot{M}}_2={\dot{M}}_{\{3,6,9\}}$. The automaton starts in state 0, then a transition $0\stackrel{\mathtt{y}}{\to }2$ is performed.

$$\begin{array}{cc}\hfill m\left(\right\{3,6,9\},2)& =\nu \left(\right\{3,6,9\left\}\right)\cap m\left(2\right)=\{d,f,g\}\cap \left\{d\right\}=\left\{d\right\}\hfill \\ \hfill p\left(\right\{3,6,9\},2)& =\nu \left(\right\{3,6,9\left\}\right)\cap p\left(2\right)=\{d,f,g\}\cap \left\{d\right\}=\left\{d\right\}\hfill \end{array}$$

Then a transition $2\stackrel{\$}{\to }6$ is performed.

$$\begin{array}{cc}\hfill m\left(\right\{3,6,9\},6)& =p\left(\right\{3,6,9\},2)\cap m\left(6\right)=\left\{d\right\}\cap \varnothing =\varnothing \hfill \\ \hfill p\left(\right\{3,6,9\},6)& =p\left(\right\{3,6,9\},2)\cap p\left(6\right)=\left\{d\right\}\cap \varnothing =\varnothing \hfill \end{array}$$

As already mentioned, context-aware scanning allows some of the specifications where the scanner automata are not pairwise disjoint or pairwise prefix disjoint, to be parsed deterministically. In this example, the languages of $M_c$ and $M_e$ are not disjoint, since they both match $\mathtt{x}$, which results in $\left|m\right(1\left)\right|>1$. However, there is no $\mathbf{P}\in \Pi$ and $\dot{q}\in \dot{\mathbf{Q}}$, such that $m(\mathbf{P},\dot{q})=\{c,e\}$. This is easy to verify, $\Pi =\left\{\right\{0\},\{8\},\{8,5\},\{8,5,2\},\{3,6,9\},\{4\},\{4,1\},\{7\},$$\{7,1\},\left\{10\right\},\{10,1\}\}$. If that were not the case, there would be a lexical conflict. The possibility of such conflicts occurring was already identified by Tomita [15,16]; only a minor modification is required to the RNGLR parsing algorithm. The idea was later termed Schrödinger’s token by Aycock and Horspool [3]. In short, the algorithm performs the actions for all $t\in m(\mathbf{P},\dot{q})$.

The languages of automata $M_c$ and $M_g$ are disjoint; however, they are not prefix-disjoint, since $\mathtt{x}$ is a prefix of $\mathtt{x}\mathtt{x}$. There is again no $\mathbf{P}\in \Pi$ and $\dot{q}\in \dot{\mathbf{Q}}$, such that $m(\mathbf{P},\dot{q})=\{c,g\}$. If that were not the case, there would be a lexical conflict. This type of lexical conflict is much harder to resolve since the recognized lexemes are of different lengths. The scanner would end up in two positions after matching $M_c|M_g$. Subsequently, the scanner and the parser would need to split to handle both choices as proposed by Begel and Graham [14]. The same lexical conflict occurs if the languages of automata ${\left(M_t\right)}_{t\in \mathbf{T}}$ are not prefix-free. For example, the language $\{\epsilon ,\mathtt{x},\mathtt{x}\mathtt{x},\dots \}$, corresponding to $r={\mathtt{x}}^{*}$ exhibit this issue. This is usually resolved using the longest match disambiguation rule; however, this solution is not general. For example, the following specification cannot be parsed:

$$S::=ab{r}_a={\mathtt{x}}^{*}{r}_b=\mathtt{x}{r}_{\$}=\$$$

If we take a zoomed-out view of the GSS for $\Xi$ in (2), focusing only on the shift actions, as presented in Figure 9, we can notice that there is a single path between any two subclasses. This is due to the fact that scanning has been completely deterministic up until this point. At each position $\mathfrak{i}$, at most, one lexeme could be recognized, which corresponds to (at most) one lookahead symbol.

In general, scanning is non-deterministic. Multiple automata ${\left(M_t\right)}_{t\in {\mathbf{T}}_{\$}}$ can match the same lexeme, and/or multiple different lexemes can be recognized. Next, we present our extension of the context-aware scanning architecture and RNGLR parsing, which can handle the non-determinism efficiently.

4. RNGSGLR (0,1]

The algorithm presented in this section will be a recognizer. The lexical conflict resulting from multiple automata ${\left(M_t\right)}_{t\in {\mathbf{T}}_{\$}}$ matching the same lexeme can already be resolved efficiently using Schrödinger’s token [3,15,16]. The problem we tackle is the lexical conflict resulting from recognizing multiple lexemes of different lengths.

Our architecture builds on context-aware scanning, likewise, we have the automata ${\left({\dot{M}}_{\mathfrak{i}}\right)}_{\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}}$ for each position in the parse. The difference is that we consider all possible lexemes matched by each ${\dot{M}}_{\mathfrak{i}}$. Let ${\dot{q}}_{\mathfrak{i}/\mathfrak{i}}\to \dots \to {\dot{q}}_{\mathfrak{i}/\mathfrak{l}}$ be a path traced by the automaton ${\dot{M}}_{\mathfrak{i}}$. We use $\mathfrak{i}/\mathfrak{k}$ to denote that the state ${\dot{q}}_{\mathfrak{i}/\mathfrak{k}}$ belongs to ${\dot{M}}_{\mathfrak{i}}$ and that it was entered at position $\mathfrak{k}$. The state entered at position $\mathfrak{i}$ is the initial state; that is ${\dot{q}}_{\mathfrak{i}/\mathfrak{i}}={\dot{q}}_{◊}$, and the state entered at position $\mathfrak{l}$ is the last state, a state from which no more matches can be made if the automaton has one or $\mathfrak{l}=\left|\omega \right|+1$. Some of these states are final; the set of positions in which the automaton enters a final state is:

$$\phi ({\dot{q}}_{\mathfrak{i}/\mathfrak{i}}\to \dots \to {\dot{q}}_{\mathfrak{i}/\mathfrak{l}})=\{\mathfrak{k}\mid \mathfrak{k}\in \{\mathfrak{i}+1,\dots ,\mathfrak{l}\}\wedge {\dot{q}}_{\mathfrak{i}/\mathfrak{k}}\in \dot{\mathbf{F}}\}$$

We consider all lexemes ${\upsilon }_{(\mathfrak{i},\mathfrak{k})}$, for all $\mathfrak{k}\in \phi ({\dot{q}}_{\mathfrak{i}/\mathfrak{i}}\to \dots \to {\dot{q}}_{\mathfrak{i}/\mathfrak{l}})$, where each is a substring of $\omega$ that starts at $\mathfrak{i}$ and ends at $\mathfrak{k}$, such that ${\dot{q}}_{\mathfrak{i}/\mathfrak{i}}\stackrel{{\upsilon }_{(\mathfrak{i},\mathfrak{k})}}{\to }{\dot{q}}_{\mathfrak{i}/\mathfrak{k}}\in \dot{\mathbf{F}}$. Note that the start state is not considered, as there is no need to recognize the lexeme ${\upsilon }_{(\mathfrak{i},\mathfrak{i})}=\epsilon$ as it is trivially present at any position.

As a result, the automaton ${\dot{M}}_{\mathfrak{i}}$ that started scanning at position $\mathfrak{i}$ must scan alongside automata ${\left({\dot{M}}_{\mathfrak{h}}\right)}_{\mathfrak{h}\in \{0,\dots ,\mathfrak{i}-1\}}$ that started scanning at previous positions. For this reason, the scanner automaton at position $\mathfrak{i}$ in the parse ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle =({\mathring{\mathbf{Q}}}_{\mathfrak{i}},{\mathbf{\Omega }}_{\$},{\mathring{\delta }}_{\mathfrak{i}},{\mathring{q}}_{\mathfrak{i}}\in {\mathring{\mathbf{Q}}}_{\mathfrak{i}},{\mathring{\mathbf{F}}}_{\mathfrak{i}})$ is defined as:

$$\begin{array}{cc}\hfill {\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle & ={|}_{\mathfrak{k}\in \{0,\dots ,\mathfrak{i}\}}{\dot{M}}_{\mathfrak{k}}\langle {\dot{q}}_{\mathfrak{k}/\mathfrak{i}}\rangle \hfill \\ \hfill {\mathring{q}}_{\mathfrak{i}}& ={\left({\dot{q}}_{\mathfrak{k}/\mathfrak{i}}\right)}_{\mathfrak{k}\in \{0,\dots ,\mathfrak{i}\}}\hfill \end{array}$$

It is a runtime simulated union of automata ${\left({\dot{M}}_{\mathfrak{k}}\langle {\dot{q}}_{\mathfrak{k}/\mathfrak{i}}\rangle \right)}_{\mathfrak{k}\in \{0,\dots ,\mathfrak{i}\}}$. These started scanning at position $\mathfrak{k}$ and are now in state ${\dot{q}}_{\mathfrak{k}/\mathfrak{i}}$, as they already traced a path ${\dot{q}}_{\mathfrak{k}/\mathfrak{k}}\stackrel{{\psi }_{(\mathfrak{k},\mathfrak{i})}}{\to }{\dot{q}}_{\mathfrak{k}/\mathfrak{i}}$. The string ${\psi }_{(\mathfrak{k},\mathfrak{i})}$ is a prefix of the lexeme that the automaton is looking forward to recognize in the future. Each automaton ${\dot{M}}_{\mathfrak{k}}\langle {\dot{q}}_{\mathfrak{k}/\mathfrak{i}}\rangle$ is thus scanning for the not yet recognized suffix. This description is also trivially valid for the automaton ${\dot{M}}_{\mathfrak{i}}\langle {\dot{q}}_{\mathfrak{i}/\mathfrak{i}}\rangle$ added at position $\mathfrak{i}$, which is in the start state. The scanner automaton state ${\mathring{q}}_{\mathfrak{i}}$ is composed out of the states of automata in the union at position $\mathfrak{i}$.

The scanner automaton at position $\mathfrak{i}$ can also be defined recursively:

$$\begin{array}{cc}\hfill {\mathring{M}}_0\langle {\mathring{q}}_0\rangle & ={\dot{M}}_0\hfill \\ \hfill {\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle & ={\mathring{M}}_{\mathfrak{i}-1}\langle {\mathring{q}}_{\mathfrak{i}}^{\prime }\rangle |{\dot{M}}_{\mathfrak{i}}\hfill \\ \hfill {\mathring{q}}_{\mathfrak{i}}^{\prime }& ={\left({\dot{q}}_{\mathfrak{h}/\mathfrak{i}}\right)}_{\mathfrak{h}\in \{0,\dots ,\mathfrak{i}-1\}}\hfill \\ \hfill {\mathring{q}}_{\mathfrak{i}}& ={\mathring{q}}_{\mathfrak{i}}^{\prime }⌢{\dot{q}}_{\mathfrak{i}/\mathfrak{i}}\hfill \end{array}$$

The scanner automaton at position 0 is initially just the automaton ${\dot{M}}_0$. At each position $\mathfrak{i}-1$, the scanner automaton ${\mathring{M}}_{\mathfrak{i}-1}\langle {\mathring{q}}_{\mathfrak{i}-1}\rangle$ performs the transition ${\mathring{q}}_{\mathfrak{i}-1}\stackrel{a_{\mathfrak{i}}}{\to }{\mathring{q}}_{\mathfrak{i}}^{\prime }$ on the character $a_{\mathfrak{i}}\in {\mathbf{\Omega }}_{\$}$, becoming the automaton ${\mathring{M}}_{\mathfrak{i}-1}\langle {\mathring{q}}_{\mathfrak{i}}^{\prime }\rangle$. The scanner automaton ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$ is constructed by adding the automaton ${\dot{M}}_{\mathfrak{i}}$ in the runtime simulated union. The scanner automaton state ${\mathring{q}}_{\mathfrak{i}}$ contains the states of ${\mathring{q}}_{\mathfrak{i}}^{\prime }$ and additionally the start state ${\dot{q}}_{\mathfrak{i}/\mathfrak{i}}$ for ${\dot{M}}_{\mathfrak{i}}$.

As the input string $\omega \$$ is traversed, the scanner automaton traces a path ${\mathring{q}}_0\to \dots \to {\mathring{q}}_{\left|\omega \right|+1}^{\prime }$, where for all $\mathfrak{i}\in \{1,\dots ,|\omega |+1\}$, ${\mathring{q}}_{\mathfrak{i}}^{\prime }={\mathring{\delta }}_{\mathfrak{i}-1}({\mathring{q}}_{\mathfrak{i}-1},a_{\mathfrak{i}})$. The set of positions where the scanner automaton enters a final state is:

$$\phi ({\mathring{q}}_0\to \dots \to {\mathring{q}}_{\left|\omega \right|+1}^{\prime })=\bigcup _{\mathfrak{i}\in \{0,\dots ,|\omega |+1\}}\phi ({\dot{q}}_{\mathfrak{i}/\mathfrak{i}}\to \dots \to {\dot{q}}_{\mathfrak{i}/\mathfrak{l}})$$

Therefore, all possible lexemes ${\upsilon }_{(\mathfrak{i},\mathfrak{k})}$, for all $\mathfrak{i},\mathfrak{k}\in \phi ({\mathring{q}}_0\to \dots \to {\mathring{q}}_{\left|\omega \right|+1}^{\prime })\wedge \mathfrak{i}\le \mathfrak{k}$ are considered. The lexemes can overlap arbitrarily. If they do not, our architecture degenerates to context-aware scanning.

The description given so far may seem infeasible, as defined, the number of states grows with the position in the parse. However, basic algebraic properties of the union, $M=⌀|M=M|⌀=M|M$ can be used as the optimization [2].

• Once no more matches can be made by the automaton, then ${\dot{M}}_{\mathfrak{k}}\langle {\dot{q}}_{\mathfrak{k}/\mathfrak{i}}\rangle =⌀$, and the automaton can be removed from the union. Therefore, at each position $\mathfrak{i}$, the automata, where $p({\mathbf{P}}_{\mathfrak{i},0},{\dot{q}}_{\mathfrak{k}/\mathfrak{i}})=\varnothing$ are filtered out.
• If no matches were found at position $\mathfrak{i}$, it immediately means ${\mathbf{P}}_{\mathfrak{i},0}=\varnothing$, as no shift actions can be performed by the parser. As a result, ${\dot{M}}_{\mathfrak{i}}=⌀$; therefore, ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle ={\mathring{M}}_{\mathfrak{i}-1}\langle {\mathring{q}}_{\mathfrak{i}}^{\prime }\rangle |⌀$. This allows the scanner to move uninterruptedly to the next match.
• It is possible that automata are duplicates $\mathfrak{k}\ne \mathfrak{l}\wedge {\dot{M}}_{\mathfrak{k}}\langle {\dot{q}}_{\mathfrak{k}/\mathfrak{i}}\rangle ={\dot{M}}_{\mathfrak{l}}\langle {\dot{q}}_{\mathfrak{l}/\mathfrak{i}}\rangle$. This happens if they are in the same state ${\dot{q}}_{\mathfrak{k}/\mathfrak{i}}={\dot{q}}_{\mathfrak{l}/\mathfrak{i}}$, and if they can match the same set of symbols $p({\mathbf{P}}_{\mathfrak{k},0},{\dot{q}}_{\mathfrak{k}/\mathfrak{i}})=p({\mathbf{P}}_{\mathfrak{l},0},{\dot{q}}_{\mathfrak{l}/\mathfrak{i}})$. Such automata are merged, resulting in an automaton ${\dot{M}}_{\mathfrak{k},\mathfrak{l}}\langle {\dot{q}}_{\mathfrak{k},\mathfrak{l}/\mathfrak{i}}\rangle$. It is crucial that both starting positions are preserved. An example of a specification that exhibits such duplication is:

  $$S::=aS|\epsilon r_a={\mathtt{x}}^{*}{r}_{\$}=\$$$

These optimizations are not strictly required; even the most naive implementation always results in a correct parse, albeit, it invariably exhibits the worst-case behavior.

The functions $m:{\mathring{\mathbf{Q}}}_{\mathfrak{i}}^{\prime }\to \mathcal{P}\left(\right\{0,\dots ,\left|\omega \right|\}\times {\mathbf{T}}_{\$})$ and $p:{\mathring{\mathbf{Q}}}_{\mathfrak{i}}^{\prime }\to \mathcal{P}\left(\right\{0,\dots ,\left|\omega \right|\}\times {\mathbf{T}}_{\$})$ are defined as:

$$\begin{array}{cc}\hfill m\left({\mathring{q}}_{\mathfrak{i}}^{\prime }\right)& =\bigcup _{\mathfrak{h}\in \{0,\dots ,\mathfrak{i}-1\}}\left\{(\mathfrak{h},t)\mid t\in m({\mathbf{P}}_{\mathfrak{h},0},{\dot{q}}_{\mathfrak{h}/\mathfrak{i}})\right\}\hfill \\ \hfill p\left({\mathring{q}}_{\mathfrak{i}}^{\prime }\right)& =\bigcup _{\mathfrak{h}\in \{0,\dots ,\mathfrak{i}-1\}}\left\{(\mathfrak{h},t)\mid t\in p({\mathbf{P}}_{\mathfrak{h},0},{\dot{q}}_{\mathfrak{h}/\mathfrak{i}})\right\}\hfill \end{array}$$

The position $\mathfrak{h}$ is added because the recognized lexeme ${\upsilon }_{(\mathfrak{h},\mathfrak{i})}$, starts at position $\mathfrak{h}$. That means the shift actions need to be performed from ${\mathbf{U}}_{\mathfrak{h}}$.

The scanning process remains very similar to the (traditional) context-aware scanning process. At each position $\mathfrak{i}$ in the parse, the scanner recognizes a string ${\psi }_{(\mathfrak{i},\mathfrak{j})}\in {\mathbf{\Omega }}_{\$}^{+}$ in the rest of the input string ${\omega }^{\prime }\in {\mathbf{\Omega }}_{\$}^{+}$ using the automaton ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$, such that ${\omega }^{\prime }={\psi }_{(\mathfrak{i},\mathfrak{j})}{\omega }^{″}$ and ${\mathring{q}}_{\mathfrak{i}}\stackrel{{\psi }_{(\mathfrak{i},\mathfrak{j})}}{\to }{\mathring{q}}_{\mathfrak{j}}^{\prime }\in {\mathring{\mathbf{F}}}_{\mathfrak{i}}$. Note that the scanner stops only when a match is found at position $\mathfrak{j}$. This is the same as in (traditional) context-aware scanning, except that multiple lexemes ${\upsilon }_{(\mathfrak{k},\mathfrak{j})}$ are recognized for each automaton ${\left({\dot{M}}_{\mathfrak{k}}\right)}_{\mathfrak{k}\in \{0,\dots ,\mathfrak{i}\}}$, which enters the final state. Moreover, ${\psi }_{(\mathfrak{i},\mathfrak{j})}$ might not even be a lexeme. If the only lexeme that is recognized is ${\upsilon }_{(\mathfrak{i},\mathfrak{j})}$, then our architecture degenerates to context-aware scanning for that portion of the input string. Each lexeme ${\upsilon }_{(\mathfrak{k},\mathfrak{j})}$ is recognized in $\left|\phi \right({\mathring{q}}_{\mathfrak{k}}\to \dots \to {\mathring{q}}_{\mathfrak{j}}^{\prime }\left)\right|$ steps. That means that the scanning for each lexeme is suspended and resumed for each match between the positions $\mathfrak{k}$ and $\mathfrak{j}$. The recognizer is passed $m\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)$ and $p\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)$. The process is then repeated for $\mathfrak{j}$ and ${\omega }^{″}$ using ${\mathring{M}}_{\mathfrak{j}}\langle {\mathring{q}}_{\mathfrak{j}}\rangle ={\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{j}}^{\prime }\rangle |{\dot{M}}_{\mathfrak{j}}$.

Thus far, we only covered our extension of context-aware scanning, so let us motivate the needed modifications to the RNGLR parsing with an example:

$$\begin{array}{cc}\hfill S& ::=bcd|Ae\hfill \\ \hfill A& ::=b\hfill \end{array}\begin{array}{cc}\hfill r_b& =\mathtt{x}\hfill \\ \hfill r_c& =\mathtt{y}\hfill \end{array}\begin{array}{cc}\hfill r_d& =\mathtt{z}\hfill \\ \hfill r_e& =\mathtt{y}\mathtt{z}\hfill \end{array}\begin{array}{cc}\hfill r_{\$}& =\$\hfill \\ \hfill \end{array}$$

(3)

The handle-finding automaton, constructed using $G^{\prime }$ in (3), is presented in Figure 10, and the scanner automaton is presented in Figure 11. Notice that all ${\left(M_t\right)}_{t\in \mathbf{T}}$ are prefix-free, $M_c$ and $M_e$ are not prefix-disjoint. Recognizing the input string $\omega =\mathtt{x}\mathtt{y}\mathtt{z}$ results in the GSS and the scanner trace in Figure 12 and Figure 13 respectively.

The valid lookahead sets for each handle-finding automaton state are $\nu \left(0\right)=\left\{b\right\}$ and $\nu \left(1\right)=\nu \left(3\right)=\nu \left(6\right)=\left\{\$\right\}$ and $\nu \left(2\right)=\left\{e\right\}$ and $\nu \left(4\right)=\{c,e\}$ and $\nu \left(5\right)=\left\{d\right\}$. They are used to determine the valid lookahead sets $\nu \left({\mathbf{P}}_{\mathfrak{i},0}\right)$ at each position $\mathfrak{i}$ based on the sets of states ${\mathbf{P}}_{\mathfrak{i},0}$, both of which are given in 

Table 2. The sets of states ${\mathbf{P}}_{\mathfrak{i},0}$ and the valid lookahead sets $\nu \left({\mathbf{P}}_{\mathfrak{i},0}\right)$ for the specification $\Xi$ in (3).

$\mathfrak{i}$	$P_{\mathfrak{i},0}$	$\mathit{\nu }\left(P_{\mathfrak{i},0}\right)$
0	0	b
1	4	$c,e$
2	5	d
3	$3,6$	$

.

We start with ${\mathring{M}}_0\langle 0\rangle ={\dot{M}}_0\langle 0\rangle$, and scan until the first match $m\left(\right(2\left)\right)=\left\{\right(0,b\left)\right\}$, tracing a path $\left(0\right)\stackrel{\mathtt{x}}{\to }\left(2\right)$. The scanning continues with ${\mathring{M}}_1\langle (2,0)\rangle ={\dot{M}}_0\langle 2\rangle |{\dot{M}}_1\langle 0\rangle$ to the next match $m\left(\right(5,1\left)\right)=\left\{\right(1,c\left)\right\}$, tracing a path $(2,0)\stackrel{\mathtt{y}}{\to }(5,1)$. Now a reduce action can be performed; however, notice that the item $(A::=b•e)$ has e as the lookahead symbol, which will be recognized in two steps. At this point, we already covered the first step; however, we still need to cover the second for e to be matched. We could scan ahead; however, this would mean the scanner and the parser would need to split. Instead, we choose to work with the information we have at this point. We can use $p\left(\right(5,1\left)\right)=\left\{\right(1,e\left)\right\}$, which tells us that we can still expect to match e and make the reduction based on this information. At worst, this means a superfluous reduction will be made, which will be ignored in the future. Since $\frac{1}{2}$ of the lexeme have been recognized, we call this approach the fractional lookahead. The automaton ${\dot{M}}_0\langle 5\rangle$ enters a dead state, so it can be removed. We continue scanning with ${\mathring{M}}_2\langle (1,0)\rangle ={\dot{M}}_1\langle 1\rangle |{\dot{M}}_2\langle 0\rangle$ to the next match $m\left(\right(3,3\left)\right)=\left\{\right(1,e),(2,d\left)\right\}$, tracing a path $(1,0)\stackrel{\mathtt{z}}{\to }(3,3)$ The first thing to note is that $\left|m\right((3,3)\left)\right|>1$. This lexical conflict occurs because $\mathcal{L}\left({\dot{M}}_1\langle 1\rangle \right)\asymp \mathcal{L}\left({\dot{M}}_2\langle 0\rangle \right)$ does not hold. As already mentioned, this type of lexical conflict can be resolved by performing the actions for both e and d. In this case, a shift action is performed for both symbols. The second thing to note is that, to perform a shift action using e, it needs to be performed from ${\mathbf{U}}_1$. In traditional RNGLR parsing a shift action can only be performed from the previous subclass, which is ${\mathbf{U}}_2$; therefore, the parsing algorithm needs to be modified. We continue scanning with ${\mathring{M}}_3\langle (3,3,0)\rangle ={\dot{M}}_1\langle 3\rangle \left|{\dot{M}}_2\langle 3\rangle \right|{\dot{M}}_3\langle 0\rangle$ to the next match $m\left(\right(5,5,4\left)\right)=\left\{\right(3,\$\left)\right\}$, tracing a path $(5,1,0)\stackrel{\$}{\to }(5,5,4)$. The reduce action is performed for S and parsing concludes successfully.

The scanner trace is a visual presentation of the path ${\mathring{q}}_0\to \dots \to {\mathring{q}}_{\left|\omega \right|+1}^{\prime }$. The path is, in this case, concretely:

$$\left(0\right)\stackrel{\mathtt{x}}{\to }(2,0)\stackrel{\mathtt{y}}{\to }(5,1,0)\stackrel{\mathtt{z}}{\to }(3,3,0)\stackrel{\$}{\to }(5,5,4)$$

The scanner trace is not a data structure, at each point in the parse, only a single column is present in the memory. At each position $\mathfrak{i}\in \{0,1,2,3\}$, the column represents the state ${\mathring{q}}_{\mathfrak{i}}$. At each position $\mathfrak{j}\in \{1,2,3,4\}$, the column without the added start state represents the state ${\mathring{q}}_{\mathfrak{j}}^{\prime }$. The rows represent the paths ${\dot{q}}_{\mathfrak{i}/\mathfrak{i}}\to \dots \to {\dot{q}}_{\mathfrak{i}/\mathfrak{l}}$ traced by each ${\left({\dot{M}}_{\mathfrak{i}}\right)}_{\mathfrak{i}\in \{0,1,2,3\}}$. Note that ${\dot{M}}_1$ traced the path $0\to 1\to 3\to 5$, where $\phi (0\to 1\to 3\to 5)=\{1,2\}$. That is, the automaton entered the final state twice and two lexemes were recognized as a result. This occurred due to the fact that the lookahead symbols $\nu \left({\mathbf{P}}_{1,0}\right)=\{c,e\}$ correspond to $M_c$ and $M_e$, the languages of which are not prefix-disjoint since $\mathtt{y}$ is a prefix of $\mathtt{y}\mathtt{z}$.

If we take a zoomed-out view of the GSS for $\Xi$ in (3), focusing only on the shift actions, as presented in Figure 14, we can see that there can be more than one path between any two subclasses. The substring $\mathtt{y}\mathtt{z}$ can be interpreted in two ways, either as e or c followed by d. In general, the shift actions alone now form a directed acyclic graph. The graph has a single path between any two subclasses exactly when our architecture degenerates to context-aware scanning.

The required modifications to the RNGLR parsing algorithm are:

• When more than one lookahead symbol is matched, $\left|m\right({\mathring{q}}_{\mathfrak{j}}^{\prime }\left)\right|>1$, the actions are performed for all of them [3,15,16].
• At position $\mathfrak{i}$, it must be possible to perform a shift action from any ${\left({\mathbf{U}}_{\mathfrak{k}}\right)}_{\mathfrak{k}\in \{0,\dots ,\mathfrak{i}\}}$.
• The reduce actions are selected based on a prediction $p\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)$, if the entire lexeme is not recognized at position $\mathfrak{j}$, which we call fractional lookahead.

The algorithm works as follows. At each position $\mathfrak{i}$ in the parse, a string ${\psi }_{(\mathfrak{i},\mathfrak{j})}\in {\mathbf{\Omega }}_{\$}^{+}$ is recognized using ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$. Instead of a single lookahead symbol $t_{\mathfrak{j}}$, there are now two sets of lookahead symbols, ${\mathbf{B}}_{\mathfrak{i}}$ and ${\mathbf{\Lambda }}_{\mathfrak{i}}$. The set ${\mathbf{B}}_{\mathfrak{i}}=m\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)$ contains the lookahead symbols and the accompanying starting positions for shift actions. The set ${\mathbf{\Lambda }}_{\mathfrak{i}}=\{t\mid (\mathfrak{k},t)\in p\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)\wedge \mathfrak{k}=\mathfrak{i}\}$ contains the lookahead symbols for the reduce actions. The lexemes for reduce actions can only start at position $\mathfrak{i}$; thus, the others are removed. Instead of just lookahead symbols that were matched, the set also contains those that can still be matched continuing from ${\mathring{q}}_{\mathfrak{j}}^{\prime }$. This is essentially how the fractional lookahead is implemented. The vertices $w\in {\mathbf{U}}_{\mathfrak{i}}$, such that $\exists t\in {\mathbf{\Lambda }}_{\mathfrak{i}},\mathrm{REDUCE}(A,|\gamma \left|\right)\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),t)$ are identified. The reduce actions are processed exactly as in RNGLR parsing. Then, for each $(\mathfrak{k},t)\in {\mathbf{B}}_{\mathfrak{i}}$ the vertices $w\in {\mathbf{U}}_{\mathfrak{k}}$, such that $\exists (\mathfrak{k},t)\in {\mathbf{B}}_{\mathfrak{i}},\mathrm{SHIFT}\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),t)$ are identified. The shift actions are processed exactly as in RNGLR parsing. Then the process is repeated for ${\omega }^{″}$ and ${\mathbf{U}}_{\mathfrak{j}}$ and ${\mathring{M}}_{\mathfrak{j}}\langle {\mathring{q}}_{\mathfrak{j}}\rangle ={\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{j}}^{\prime }\rangle |{\dot{M}}_{\mathfrak{j}}$. If there exists $w\in {\mathbf{U}}_{\left|\omega \right|}$, such that $\exists t\in {\mathbf{\Lambda }}_{\mathfrak{i}},\mathrm{ACCEPT}\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),t)$, the parsing is completed successfully.

4.1. Support for Nullable RE

To support all possible $\Xi$, the case when $\epsilon \in \mathcal{L}\left(r_t\right)$, where $t\in \mathbf{T}$, needs to be considered as well. That means the symbols $t\in \mathbf{T}$ can be nullable as well, $t\Rrightarrow \epsilon$. The solution is to use $\mathrm{FIRST}\left(t\right)=\{\epsilon ,t\}$ if $\epsilon \in \mathcal{L}\left(r_t\right)$ when computing the $\mathrm{FIRST}$ sets. In the case of shift actions, the lookahead traditionally always coincides with the symbol to be shifted; thus, no special consideration is needed [1]. When the terminal symbols can be nullable, this is no longer the case. For each state $s\in \mathbf{S}$, the shift items $(A::=\gamma •t\eta l)\in s$ are converted into $\{(q,t)\mapsto \mathrm{SHIFT}\}\cup \{(q,l^{\prime })\mapsto \epsilon \text{-}\mathrm{SHIFT}t\mid l^{\prime }\in \mathrm{FIRST}\left(t\eta l\right)\}$. The $\epsilon$-shift actions need to be differentiated because they are performed within ${\mathbf{U}}_{\mathfrak{i}}$. The definition of valid lookahead sets remains the same. That means $\epsilon$-shift actions must be considered as well.

The scanning process remains exactly the same. There is no need to recognize the lexeme ${\upsilon }_{(\mathfrak{i},\mathfrak{i})}=\epsilon$ as it is trivially present at any position $\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}$. The parsing process needs to be modified as follows: The reduce actions are processed in the same way as before. Additionally, at each position $\mathfrak{i}$ in the parse, the vertices $w\in {\mathbf{U}}_{\mathfrak{i}}$ such that $\exists l\in {\mathbf{\Lambda }}_{\mathfrak{i}},\epsilon \text{-}\mathrm{SHIFT}t\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),l)$ are identified. In the case of $\epsilon$-shift actions for the vertex $w\in {\mathbf{U}}_{\mathfrak{i}}$, a vertex $u\in {\mathbf{U}}_{\mathfrak{i}}$ is either found or created and the vertices are connected with an edge $w\leftarrow u$. The $\epsilon$-shift actions are processed in exactly the same way as  $\epsilon$-reduce actions. Since $\epsilon$-shift actions are performed within ${\mathbf{U}}_{\mathfrak{i}}$, this means new vertices can be created that need to be processed. For this reason, reduce and $\epsilon$-shift actions are processed in a common loop until there are no possible actions left. The $\epsilon$-shift actions can connect $w\in {\mathbf{U}}_{\mathfrak{i}}$ to an existent vertex $w^{\prime }\in {\mathbf{U}}_{\mathfrak{i}}$, which was created as a result of a shift action starting from ${\mathbf{U}}_{\mathfrak{h}\in \{0,\dots ,\mathfrak{i}-1\}}$. Therefore, for all such $w^{\prime }$, the reduce actions, where $\left|\gamma \right|>0$, must be processed before processing $\epsilon$-shift actions. Otherwise, the reduction step will be applied twice, once using the short-circuiting reduce action from w or one of its descendants, and once more using a reduce action from $w^{\prime }$. The shift actions are processed in the same way as before.

To demonstrate the required modifications, consider the following specification:

$$\begin{array}{cc}\hfill S& ::=dA|c\hfill \\ \hfill r_c& ={\mathtt{x}}^{*}\hfill \end{array}\begin{array}{cc}\hfill A& ::=deBe\hfill \\ \hfill r_d& =\mathtt{x}\hfill \end{array}\begin{array}{cc}\hfill B& ::=\epsilon \hfill \\ \hfill r_e& =\mathtt{x}|\epsilon \hfill \end{array}\begin{array}{c}\hfill \\ \hfill r_{\$}=\$\end{array}$$

(4)

The handle-finding automaton, constructed using $G^{\prime }$ in (4), is presented in Figure 15, and the scanner automaton is presented in Figure 16. The handle-finding automaton results in the ${\mathcal{T}}_A$ and ${\mathcal{T}}_G$ presented in 

Table 3. The action table ${\mathcal{T}}_A$ for the specification $\Xi$ in (4).

	c	d	e	$
0	s	s		$\epsilon \text{-}\mathrm{S}c,\mathrm{R}(S,0),\mathrm{A}$
1				a
2				$\mathrm{R}(S,1)$
3		s
4				$\mathrm{R}(S,2)$
5			$\epsilon \text{-}\mathrm{S}e,\mathrm{S}$	$\epsilon \text{-}\mathrm{S}e,\mathrm{R}(A,1)$
6			$\mathrm{R}(B,0)$	$\mathrm{R}(B,0),\mathrm{R}(A,2)$
7			S	$\epsilon \text{-}\mathrm{S}e,\mathrm{R}(A,3)$
8				$\mathrm{R}(A,4)$

and

Table 4. The goto table ${\mathcal{T}}_G$ for the specification $\Xi$ in (4).

	S	A	B	c	d	e	$
0	1			2	3
1
2
3		4			5
4
5						6
6			7
7						8
8

respectively. The productions that are right nullable are $S^{\prime }::=S$ and $S::=c$ and $A::=deBe$. In contrast to RNGLR parsing [17], the nullable part can contain both terminals and non-terminals. The ${\mathcal{T}}_G$ remains unmodified; however, it is still presented for completeness. Recognizing the input string $\omega =\mathtt{x}\mathtt{x}\mathtt{x}$ results in the GSS and the scanner trace in Figure 17 and Figure 18, respectively.

The valid lookahead sets for each handle-finding automaton state are $\nu \left(0\right)=\{c,d,\$\}$ and $\nu \left(3\right)=\left\{d\right\}$ and $\nu \left(5\right)=\nu \left(6\right)=\nu \left(7\right)=\{e,\$\}$ and $\nu \left(1\right)=\nu \left(2\right)=\nu \left(4\right)=\nu \left(8\right)=\left\{\$\right\}$. The valid lookahead set for state 5 is a result of $\mathrm{FIRST}\left(eBe\$\right)=\{e,\$\}$, since $e\Rrightarrow \epsilon$ and $B\Rrightarrow \epsilon$. They are used to determine the valid lookahead sets $\nu \left({\mathbf{P}}_{\mathfrak{i},0}\right)$ at each position $\mathfrak{i}$ based on the sets of states ${\mathbf{P}}_{\mathfrak{i},0}$, which in turn, are used to determine the sets ${\mathbf{\Lambda }}_{\mathfrak{i}}$ and ${\mathbf{B}}_{\mathfrak{i}}$. These are given in 

Table 5. The sets of states ${\mathbf{P}}_{\mathfrak{i},0}$, the valid lookahead sets $\nu \left({\mathbf{P}}_{\mathfrak{i},0}\right)$, and the sets ${\mathbf{\Lambda }}_{\mathfrak{i}}$ and ${\mathbf{B}}_{\mathfrak{i}}$ for the specification $\Xi$ in (4).

$\mathfrak{i}$	$P_{\mathfrak{i},0}$	$\mathit{\nu }\left(P_{\mathfrak{i},0}\right)$	${\Lambda }_{\mathfrak{i}}$	$B_{\mathfrak{i}}$
0	0	$c,d,\$$	$c,d$	$c,d$
1	2, 3	$d,\$$	d	$c,d$
2	2, 5	$e,\$$	e	$c,e$
3	2, 6, 8	$e,\$$	$	$

.

In ${\mathbf{U}}_3$ a short-circuiting reduce action $\mathrm{REDUCE}(A,2)$ is applied from state 6, and a reduce action $\mathrm{REDUCE}(A,4)$ is applied from state 8. An $\epsilon$-shift action is applied from state 7 to state 8, adding an additional descendant. The reduce action from state 8 must be applied before the edge $7\leftarrow 8$ is created, otherwise both $\mathrm{REDUCE}(A,2)$ and $\mathrm{REDUCE}(A,4)$ are applied down the path $3\leftarrow 5\leftarrow 6\leftarrow 7\leftarrow 8$. This formation occurs in the GSS, because there are two possible interpretations for $eBe$, either the first e corresponds to $\mathtt{x}$ and the second to $\epsilon$, or vice versa.

If we take a zoomed-out view of the GSS for $\Xi$ in (4), focusing only on the shift actions, as presented in Figure 19, in addition to there being multiple paths between any two subclasses, there are loops due to $\epsilon$-shift actions.

4.2. Discussion

The solution is to perform all possible traversals of the scanner automata in addition to all possible traversals of the handle-finding automaton. In our architecture, the processes remain conceptually synchronized, all reduce actions are performed before shifting the next input symbols, and all the processes move to the next position at the same time. On top of that, the scanner automata remain synchronized as well, as they all scan the next character at the same time.

The fractional lookahead arises as a direct consequence of the requirement to preserve the synchronization. At position $\mathfrak{i}$ in the parse, when scanning using ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$, it is likely that multiple matches can be found by ${\dot{M}}_{\mathfrak{i}}$. Moreover, it is entirely possible that the next match will instead be found by any of ${\left({\dot{M}}_{\mathfrak{h}}\langle {\dot{q}}_{\mathfrak{h}/\mathfrak{i}}\rangle \right)}_{\mathfrak{h}\in \{0,\dots ,\mathfrak{i}-1\}}$ and not ${\dot{M}}_{\mathfrak{i}}$. We are aware of the following alternative solutions:

• Split the scanner and the parser; in that way, one part can remain at the current position and the other can continue with the scanning. This was proposed by Begel and Graham [14].
• Continue scanning, and when the matches are found, backtrack to the current position. This was conjectured by Keynes. This would be inefficient, due to repeated scanning and backtracking [8].

When either is used, the synchronization is lost. As already mentioned, in our case, no additional scanning is performed, which preserves the synchronization, and instead the reduce actions are selected based on the predictions made from the recognized prefix ${\psi }_{(\mathfrak{i},\mathfrak{j})}$ of the lexemes. In other words, only a fraction of the lexemes is recognized at this point. In the worst-case, $|{\psi }_{(\mathfrak{i},\mathfrak{j})}|=1$, since at least one character must be scanned before a match is found. In the best case, ${\psi }_{(\mathfrak{i},\mathfrak{j})}$ is a lexeme. That happens when our architecture degenerates to context-aware scanning, and, in that case, the length of the lookahead is 1. In general, the length of the lookahead is a fraction $\frac{|{\psi }_{(\mathfrak{i},\mathfrak{j})}|}{|{\upsilon }_{(\mathfrak{i},\mathfrak{k})}|}$, where $\mathfrak{k}\in \phi ({\dot{q}}_{\mathfrak{i}/\mathfrak{i}}\to \dots \to {\dot{q}}_{\mathfrak{i}/\mathfrak{l}})$, and is, therefore, on the interval $(0,1]$. Note that the fraction varies based on which lexeme is considered; however, the value is always on the said interval.

Our architecture is flexible enough, in that the automaton ${\dot{M}}_{{\mathbf{T}}_{\$}}$ from the traditional parsing architecture could be used directly. The problem is that, in the case of lexical conflicts, matches would be found at position $\mathfrak{j}$, and then no shift actions could be performed using the corresponding lookahead symbols. As a result, the efficiency of the algorithm would be worse for the lexically ambiguous specifications. Additionally, superfluous matches would result in shorter ${\psi }_{(\mathfrak{i},\mathfrak{j})}$ and, thus, less lookahead.

A similar issue arises when using weaker handle-finding automaton construction algorithms, such as LALR or SLR [8]. In those cases, the valid lookahead sets ${\left(\nu \left(s\right)\right)}_{s\in \mathbf{S}}$ can contain additional symbols that cannot result in a sentential form. That means an invalid match can be found in certain cases. An example of a specification that exhibits this issue when used in combination with LALR is:

$$\begin{array}{cc}\hfill S& ::=Ab\hfill \\ \hfill A& ::=cAd|e\hfill \end{array}\begin{array}{cc}\hfill r_b& ={\mathtt{y}}^{*}\hfill \\ \hfill r_c& =\mathtt{x}\hfill \end{array}\begin{array}{cc}\hfill r_d& =\mathtt{y}\hfill \\ \hfill r_e& =\mathtt{z}\hfill \end{array}\begin{array}{c}\hfill r_{\$}=\$\\ \hfill \end{array}$$

If an invalid match is found at position $\mathfrak{i}$, then ${\mathbf{U}}_{\mathfrak{i}}=\varnothing$. In RNGLR parsing, it is an indication that the parsing concluded unsuccessfully; however, in the case of our architecture, ${\mathring{M}}_{\mathfrak{h}}\langle {\mathring{q}}_{\mathfrak{i}}^{\prime }\rangle$ can still find a match. Therefore, parsing concludes unsuccessfully if and only if ${\mathring{M}}_{\mathfrak{h}}\langle {\mathring{q}}_{\mathfrak{i}}^{\prime }\rangle =⌀$.

Lexical conflicts can occur at each position $\mathfrak{i}$, for the same reasons as in context-aware scanning, and additionally if the languages of automata ${\left({\dot{M}}_{\mathfrak{k}}\langle {\dot{q}}_{\mathfrak{k}/\mathfrak{i}}\rangle \right)}_{\mathfrak{k}\in \{0,\dots ,\mathfrak{i}\}}$ are not pairwise disjoint and pairwise prefix-disjoint.

The worst-case time complexity of our architecture is $O\left(\right|\mathbf{S}\left|·\right|\omega {|}^{|\widehat{\gamma }|+1})$, where $|\widehat{\gamma }|$ is the length of the longest right-hand-side of a production rule in the grammar. The reasons for this are as follows:

• There can be at most $\left|\mathbf{S}\right|$ vertices in any ${\mathbf{U}}_{\mathfrak{i}}$, since each vertex is labeled with a distinct handle-finding automaton state.
• Each vertex can have at most $\mathfrak{i}$ successors. This is due to the fact that, in the worst-case, the handle-finding automaton can start searching for $\gamma$ of some production rule $A::=\gamma \eta$, where $\eta \stackrel{*}{\Rrightarrow }\epsilon$, in each ${({\mathbf{U}}_{\mathfrak{h}})}_{\mathfrak{h}\in \{0,\dots ,\mathfrak{i}-1\}}$, and then perform $\mathfrak{i}$ reduction steps to the same vertex in ${\mathbf{U}}_{\mathfrak{i}}$.
• The number of steps required to find the paths $v\stackrel{\gamma }{\leftarrow }w$ of length $\left|\gamma \right|$, where $w\in {\mathbf{U}}_{\mathfrak{i}}$ and $v\in {\mathbf{U}}_{\mathfrak{h}}$, is limited by the number of ways the intervening input symbols can be divided among $\left|\gamma \right|$ symbols, $\left(\genfrac{}{}{0pt}{}{\mathfrak{i}-\mathfrak{h}-1}{\left|\gamma \right|-1}\right)$. Summing over all possible final positions $\mathfrak{h}\in \{0,\dots ,\mathfrak{i}-1\}$ results in $O({\mathfrak{i}}^{\left|\gamma \right|})$ steps.
• The shift actions at position $\mathfrak{i}$ can be performed from any subclass ${({\mathbf{U}}_{\mathfrak{k}})}_{\mathfrak{k}\in \{0,\dots ,\mathfrak{i}\}}$. The ${\mathbf{U}}_{\mathfrak{j}}$ can only contain $\left|\mathbf{S}\right|$ vertices at most and $\mathfrak{i}$ actions can be performed for each vertex. Therefore, the shift actions are processed in $O(|\mathbf{S}|·\mathfrak{i})$ steps.
• The automaton ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$ can be composed out of at most $\mathfrak{i}$ automata ${({\dot{M}}_{\mathfrak{k}})}_{\mathfrak{k}\in \{0,\dots ,\mathfrak{i}\}}$. In the worst-case, a match is found after scanning a single character $a\in {\mathbf{\Omega }}_{\$}$, $|{\psi }_{(\mathfrak{i},\mathfrak{j})}|=1$. Thus, $\mathfrak{i}$ steps are needed to perform the transitions on a for all $\mathfrak{i}$ of them. The worst-case time complexity of each ${\dot{M}}_{\mathfrak{i}}$ is $O(log|{\mathbf{T}}_{\$}\left|·\right|\omega |)$. Summing over all positions $\mathfrak{i}\in \{0,\dots ,|\omega |+1\}$ gives the worst-case time complexity of the scanner alone, $O(log|{\mathbf{T}}_{\$}\left|·\right|\omega {|}^2)$.
• Each reduce action at position $\mathfrak{i}$, in addition to the steps required to find the paths, requires at most $\mathfrak{i}$ steps, as the paths can end in at most $\mathfrak{i}$ vertices. The processing is, therefore, completely dominated by the search for the paths. The worst-case is when $\left|\gamma \right|=|\widehat{\gamma }|$.
• The reduce actions at position $\mathfrak{i}$ are processed in $O(|\mathbf{S}|·{\mathfrak{i}}^{|\widehat{\gamma }|})$ steps. For each newly created vertex $w\in {\mathbf{U}}_{\mathfrak{i}}$, in the worst-case, $O({\mathfrak{i}}^{|\widehat{\gamma }|})$ steps are required. For each edge $w^{\prime }\leftarrow w$, where $w^{\prime }\in {\mathbf{U}}_{\mathfrak{h}\in \{0,\dots ,\mathfrak{i}-1\}}$, connected to an existing vertex w, in the worst-case, $O({\mathfrak{i}}^{|\widehat{\gamma }|-1})$ steps are required, as there is only one way to reach $w^{\prime }$ from w. In the worst-case $\mathfrak{i}$, such connections can be made; therefore, $O(|\mathbf{S}|·{\mathfrak{i}}^{|\widehat{\gamma }|})$ steps are required in total.
• Summing over all positions $\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}$ gives $O(|\mathbf{S}\left|·\right|\omega {|}^{|\widehat{\gamma }|+1})$.

However, usually $\zeta \ll \left|\omega \right|$, where $\zeta =|\phi ({\mathring{q}}_0\to \dots \to {\mathring{q}}_{\left|\omega \right|+1}^{\prime })|$. The recognizer performs $O(|\mathbf{S}|·{\zeta }^{|\widehat{\gamma }|+1})$ and the scanner $O(log|{\mathbf{T}}_{\$}\left|·\zeta ·\right|\omega |)$ steps. Depending on the grammar, large segments of the input string can be processed by the scanner, and the recognizer is only involved when a match is found. The worst-case space complexity is $O(|\mathbf{S}\left|·\right|\omega {|}^2)$, due to the size of the GSS. At each position $\mathfrak{i}$, there can be at most $\left|\mathbf{S}\right|$ vertices, and each vertex can have $\mathfrak{i}$ successors (edges). Summing over all positions $\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}$ gives $O(|\mathbf{S}\left|·\right|\omega {|}^2)$ [16,17,35]. The analysis is based on the analysis of GLR by Kipps; refer to [35] for additional rationale.

4.3. Algorithm

The scanning algorithm is very similar to the (traditional) context-aware scanning algorithm, except that at each position $\mathfrak{i}$ in the parse, ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$ is used as the scanner automaton and after it traces a path ${\mathring{q}}_{\mathfrak{i}}\to \dots \to {\mathring{q}}_{\mathfrak{j}}^{\prime }$, both $m\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)$ and $p\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)$ are passed to the recognizer. We implement ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$ as a multi-map using ${\dot{q}}_{\mathfrak{k}/\mathfrak{i}}$ and $p({\mathbf{P}}_{\mathfrak{k},0},{\dot{q}}_{\mathfrak{k}/\mathfrak{i}})$ as keys:

$$\bigcup _{\mathfrak{k}\in \{0,\dots ,\mathfrak{i}\}}\{{\dot{M}}_{\mathfrak{k}}\langle {\dot{q}}_{\mathfrak{k}/\mathfrak{i}}\rangle \mapsto \mathfrak{k}\}$$

That way, the automata that can no longer find a match can be filtered out and the duplicates can be merged while preserving the start positions.

The parsing algorithm will be presented in the imperative form. For this reason, we use mutable references to sets for the GSS $\Gamma =(\mathcal{V},\mathcal{E})$ and the subclasses ${\left({\mathcal{U}}_{\mathfrak{i}}\right)}_{\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}}$. The presentation of the algorithm mainly follows that by Kipps [16,35] and Scott [17]. The actor is split into three functions actor, reduce_actor, shift_actor. Each one processes the actions directly by calling the corresponding processing functions reduce and shift. This allows us to specify precisely which additional actions need to be considered when a new vertex is added. In the parse function, we must first process the reduce actions, where $\left|\gamma \right|>0$, for vertices that are in ${\mathcal{U}}_{\mathfrak{i}}$ at the start. Only then can $\epsilon$-reduce and $\epsilon$-shift actions be processed.

The multi-map ${\mathcal{B}}_{\mathfrak{i}}:{\mathbf{T}}_{\$}\to \mathcal{P}\left(\mathbf{V}\right)$ maps the lookahead symbols t to vertices w, such that $\mathrm{SHIFT}\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),t)$. There is one for each ${\left({\mathcal{U}}_{\mathfrak{i}}\right)}_{\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}}$. They allow quick identification of the vertices in each subclass for which a shift action needs to be performed. These data structures are not strictly needed, since the vertices can be identified by just using ${\mathcal{U}}_{\mathfrak{i}}$; however, a search is needed in that case. The worst-case space complexity of these data structures is $O\left(\right|{\mathbf{T}}_{\$}\left|·\right|\mathbf{S}\left|·\right|\omega \left|\right)$, as there can be at most $|{\mathbf{T}}_{\$}\left|·\right|\mathbf{S}|$ entries in each, and there is one at each position in the parse.

The algorithm listing is given in Algorithm 1. The input automata ${({\dot{M}}_{\mathfrak{i}})}_{\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}}$ are just placeholders, as they are constructed at runtime. The real inputs are the automaton ${\dot{M}}_{{\mathbf{T}}_{\$}}$ and a precomputed map $\{s\mapsto \nu (s)\mid s\in \mathbf{S}\}$.

Algorithm 1. RNGSGLR(0,1] recognizer for character-level CFL.
input	${\mathcal{T}}_A$ action table, ${\mathcal{T}}_G$ goto table, ${({\dot{M}}_{\mathfrak{i}})}_{\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}}$ automata at each position in the parse,
	$\omega$ character input string
output	accept or reject the character input string
let rngsglr1 ${\mathcal{T}}_A$ ${\mathcal{T}}_G$ ${({\dot{M}}_{\mathfrak{i}})}_{\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}}$ $\omega$ =
let rec actor w =
if $\exists l\in {\mathbf{\Lambda }}_{\mathfrak{i}},\epsilon \text{-}\mathrm{SHIFT}t\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),l)$
then shift $\mathfrak{i}$ w t
if $\exists l\in {\mathbf{\Lambda }}_{\mathfrak{i}},\mathrm{SHIFT}\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),l)$
then ${\mathcal{B}}_{\mathfrak{i}}:={\mathcal{B}}_{\mathfrak{i}}\cup \{l\mapsto w\}$
for each $\exists l\in {\mathbf{\Lambda }}_{\mathfrak{i}},\mathrm{REDUCE}(A,0)\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),l)$ do
reduce w A 0
done
and reduce_actor w =
for each $\exists l\in {\mathbf{\Lambda }}_{\mathfrak{i}},\mathrm{REDUCE}(A,|\gamma \left|\right)\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),l)\wedge \left|\gamma \right|\ne 0$ do
for each successor node $w^{\prime }$ of w do
reduce $w^{\prime }$ A $\left|\gamma \right|$
done
done
and shift_actor () =
for each match $(\mathfrak{k},l)\in {\mathbf{B}}_{\mathfrak{i}}$ do
for each $w\in {\mathcal{B}}_{\mathfrak{k}}\left(l\right),\mathrm{SHIFT}\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),l)$ do
shift $\mathfrak{j}$ w l
done
done
and reduce $w^{\prime }$ A $\left|\gamma \right|$ =
for each vertex v that can be reached from $w^{\prime }$ along the path of length $\left|\gamma \right|-1$, or length 0 if $\left|\gamma \right|=0$ do
let s = ${\mathcal{T}}_G(\mathrm{STATE}\left(v\right),A)$ in
if $\nexists u\in {\mathcal{U}}_{\mathfrak{i}},\mathrm{STATE}\left(u\right)=s$ then
add vertex u to $\mathcal{V}$ labeled s and edge $(u,v)$ to $\mathcal{E}$
${\mathcal{U}}_{\mathfrak{i}}:={\mathcal{U}}_{\mathfrak{i}}\cup \left\{u\right\}$
actor u
if $\left|\gamma \right|\ne 0$
then reduce_actor u
else if $(u,v)\notin \mathcal{E}$ then
add edge $(u,v)$ to $\mathcal{E}$
if $\left|\gamma \right|\ne 0$ then
for each $\exists t\in {\mathbf{\Lambda }}_{\mathfrak{i}},\mathrm{REDUCE}(B,|\theta \left|\right)\in {\mathcal{T}}_A(\mathrm{STATE}\left(u\right),t)\wedge \left|\theta \right|\ne 0$ do
reduce v B $\left|\theta \right|$
done
and shift $\mathfrak{l}$ w t =
let s = ${\mathcal{T}}_G(\mathrm{STATE}\left(w\right),t)$ in
if $\nexists u\in {\mathcal{U}}_{\mathfrak{l}},\mathrm{STATE}\left(u\right)=s$ then
add vertex u to $\mathcal{V}$ labeled s and edge $(u,w)$ to $\mathcal{E}$
${\mathcal{U}}_{\mathfrak{l}}:={\mathcal{U}}_{\mathfrak{l}}\cup \left\{u\right\}$
if $\mathfrak{i}=\mathfrak{l}$
then actor u
else if $(u,w)\notin \mathcal{E}$
then add edge $(u,w)$ to $\mathcal{E}$
in
let parse () =
scan at position $\mathfrak{i}$ in $\omega \$$ using ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$ tracing ${\mathring{q}}_{\mathfrak{i}}\to {\mathring{q}}_{\mathfrak{j}}^{\prime }$
get matches ${\mathbf{B}}_{\mathfrak{i}}=m\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)$
get predictions ${\mathbf{\Lambda }}_{\mathfrak{i}}=\{t\mid (\mathfrak{k},t)\in p\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)\wedge \mathfrak{k}=\mathfrak{i}\}$
${\mathcal{B}}_{\mathfrak{i}}:=\varnothing ,{\mathcal{U}}_{\mathfrak{j}}:=\varnothing$
for each vertex $w\in {\mathcal{U}}_{\mathfrak{i}}$ do
reduce_actor w
done
for each vertex $w\in {\mathcal{U}}_{\mathfrak{i}}$ do
actor w
done
shift_actor ()
construct ${\mathring{M}}_{\mathfrak{j}}\langle {\mathring{q}}_{\mathfrak{j}}\rangle ={\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{j}}^{\prime }\rangle |{\dot{M}}_{\mathfrak{j}}$
$\mathfrak{i}:=\mathfrak{j}$
in
add the start vertex $v_{◊}$ to $\mathcal{V}$ labeled $s_{◊}$
$\mathfrak{i}:=0,{\mathcal{U}}_0:=\left\{v_{◊}\right\}$
construct ${\mathring{M}}_0\langle {\mathring{q}}_0\rangle ={\dot{M}}_0$
while $\mathfrak{i}\le \left|\omega \right|\wedge {\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle \ne ⌀$ do
parse ()
done
if $\exists w\in {\mathcal{U}}_{\left|\omega \right|},\mathrm{ACCEPT}\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),\$)$
then accept
else reject

5. Construction of the Parse Forest

In this section, we extend the recognizer to a parser. To construct the parse forest, the following considerations are needed:

• The missing nullable parts must be added to the nodes produced by short-circuiting reduce actions [17].
• Multiple leaf nodes can be created at each point in the parse, since multiple lexemes can be recognized and each can correspond to multiple lookahead symbols.

We use an approach reminiscent of that of Scott [17], which, in turn, is based on that of Rekers [36].

The parsing algorithm finds all possible right-most derivations; therefore, multiple derivation trees can be constructed for a given sentence if the grammar is ambiguous. The construction of all individual derivation trees requires an exponential amount of work, and in certain cases might not terminate, even though the worst-case time complexity of the algorithm is polynomial [15,16]. The solution is to combine the common parts; that is, the subtrees are shared if multiple trees have a common subtree. If multiple nodes that correspond to the same non-terminal symbol have subtrees, which derive the same substring, then these nodes are merged into a single packed node. The resultant data structure is called a shared packed parse forest (SPPF) [16]. The SPPF is a variant of a directed graph, where each node can have multiple distinct sets of children (in the case of packed nodes). Each set of children is represented as a dot in the diagrams. Each node is labeled by a triple $(X,\mathfrak{k},\mathfrak{l})$, such that $X\in ({\mathbf{N}}^{\prime }\cup \mathbf{T})$ and $X\stackrel{*}{\Rrightarrow }{\psi }_{(\mathfrak{k},\mathfrak{l})}$, where ${\psi }_{(\mathfrak{k},\mathfrak{l})}$ is a substring of the input string $\omega$, starting at $\mathfrak{k}$ and ending at $\mathfrak{l}$. For example, the SPPF for $\Xi$ in (3) and $\omega =\mathtt{x}\mathtt{y}\mathtt{z}$ is presented in Figure 20.

The presence of nullable productions in the grammar may result in certain sentences having infinitely many derivations. As a result, $\epsilon \text{-}\mathrm{shift}$ and $\epsilon \text{-}\mathrm{reduce}$ actions in those cases create cycles in the GSS, which, in turn, result in the cycles in the SPPF. The derivation trees can be recovered by unwinding the cycles [16,17]. An example of such a specification is $\Xi$ in (5), which is presented further below.

There is a one-to-one correspondence between the edges in the GSS and the nodes in the SPPF. The construction is performed as follows: As a shift or an $\epsilon \text{-}\mathrm{shift}$ action corresponding to t is applied from the vertex $w\in {\mathbf{U}}_{\mathfrak{k}}$, a vertex $u\in {\mathbf{U}}_{\mathfrak{l}}$ is either found or created. We search the SPPF for a node z labeled $(t,\mathfrak{k},\mathfrak{l})$. If it does not exist, it is created. The vertices are connected with an edge $w\leftarrow u$, labeled z. The node corresponds to a lexeme ${\upsilon }_{(\mathfrak{k},\mathfrak{l})}$. The lexemes should not be included in the nodes directly. Since they can overlap, their size can, in total, far exceed the size of the input string. As a reduce action corresponding to $A::=\beta$ is applied from the vertex $w\in {\mathbf{U}}_{\mathfrak{i}}$, the paths $v\stackrel{\beta }{\leftarrow }w$ of length $\left|\beta \right|$ are found. As each path is traversed we record the labels on the edges, $z_1,\dots ,z_{\left|\beta \right|}$. For each $v\in {\mathbf{U}}_{\mathfrak{k}}$ a vertex $u\in {\mathbf{U}}_{\mathfrak{i}}$ is either found or created. We search the SPPF for a node z labeled $(A,\mathfrak{i},\mathfrak{i})$ in the case of $\epsilon \text{-}\mathrm{reduce}$ actions, or $(A,\mathfrak{k},\mathfrak{i})$ in the case of reduce actions, where the node $z_1$ is labeled $(B,\mathfrak{k},\mathfrak{f})$. If it does not exist, it is created. The nodes $z_1,\dots ,z_{\left|\beta \right|}$ are then added as children to the node z if they are not already included. The vertices are connected with an edge $v\leftarrow u$ labeled z [17,36]. In the case of right nullable productions $A::=\gamma \eta$, where $\eta \stackrel{*}{\Rrightarrow }\epsilon$, the subtrees corresponding to $\eta$ are missing from the SPPF, since the reduction steps are short-circuited [17]. To remedy this, Scott [17] pre-constructed the SPPF for $\eta$ of each right nullable production $A::=\gamma \eta$ in the grammar, called the $\epsilon$-SPPF. The nodes of the $\epsilon$-SPPF are indexed, and the index is included in the reduce actions. As a short-circuiting reduce action is applied, a node $z^{\prime }$ from the $\epsilon$-SPPF corresponding to the index is appended to the nodes of each path: $z_1,\dots ,z_{\left|\gamma \right|},z^{\prime }$. The $\epsilon$-SPPF consists of nodes created using productions $A::=\beta$, where $\beta \stackrel{*}{\Rrightarrow }\epsilon$ and terminal symbols $t\Rrightarrow \epsilon$. For this reason, $\epsilon$-SPPF needs to be used when such nodes are created using $\epsilon$-reduce or $\epsilon$-shift actions, otherwise the nodes would be duplicated. Thus, the index of the $\epsilon$-SPPF node needs to be included in each of these actions as well. The $\epsilon$-SPPF nodes are just placeholders, and the appropriate SPPF nodes need to be inserted at their places after the GSS is constructed [17].

In our approach, we construct these SPPF nodes immediately. We include $\eta$ directly in the reduce actions, $\mathrm{REDUCE}(A,|\gamma |,\eta )$. To generate a subtree for all derivations of $X\stackrel{*}{\Rrightarrow }\epsilon$, where $X\in ({\mathbf{N}}^{\prime }\cup \mathbf{T})$, we search the SPPF for a node z labeled $(X,\mathfrak{i},\mathfrak{i})$. If it does not exist, it is created. For each $\kappa$, such that $X\Rightarrow \kappa$, where $\kappa \stackrel{+}{\Rrightarrow }\epsilon$, we repeat the same process for all symbols $\kappa =X_1,\dots ,X_{\left|\kappa \right|}$, and add the resultant root nodes $z_1,\dots ,z_{\left|\kappa \right|}$ as children to node z. This requires us to pass the grammar $G^{\prime }$ to the parser; however, only productions $A::=\beta$, where $\beta \stackrel{*}{\Rrightarrow }\epsilon$ are needed. Therefore, we can construct a trimmed-down version $G_{\epsilon }^{\prime }$ with only those productions. As a short-circuiting reduce action is applied, we generate the subtrees for all symbols $\eta =X_1,\dots ,X_{\left|\eta \right|}$ corresponding to all derivations of $\eta \stackrel{*}{\Rrightarrow }\epsilon$, and the resultant root nodes $z_1^{\prime },\dots ,z_{\left|\eta \right|}^{\prime }$ are appended to the nodes of each path: $z_1,\dots ,z_{\left|\gamma \right|},z_1^{\prime },\dots ,z_{\left|\eta \right|}^{\prime }$. No modifications are required to the processing of $\epsilon$-shift and $\epsilon$-reduce actions, as the nodes from SPPF are reused when generating the subtrees. The benefit of our approach is that the SPPF is constructed to completion at each point during parsing. The downside is that the tables are slightly larger since the reduce actions additionally include a string instead of a single index. However, since the worst-case time complexity of RNGLR parsing depends on the lengths of the right-hand sides, we can expect these strings to be reasonably short. Despite the fact that the construction process is not exactly the same as that of Scott [17], the resulting SPPF is structurally equivalent. Therefore, our architecture can be used as a drop-in replacement for RNGLR in combination with (traditional) context-aware scanning.

Let us demonstrate the SPPF construction with the following example:

$$\begin{array}{cc}\hfill S& ::=A|eB\hfill \\ \hfill C& ::=\epsilon \hfill \end{array}\begin{array}{cc}\hfill A& ::=dA|\epsilon \hfill \\ \hfill r_d& =\mathtt{x}|\epsilon \hfill \end{array}\begin{array}{cc}\hfill B& ::=CC\hfill \\ \hfill r_e& =\mathtt{x}\hfill \end{array}\begin{array}{c}\hfill r_{\$}=\$\\ \hfill \end{array}$$

(5)

which corresponds to the following trimmed-down the grammar of nullable productions $G_{\epsilon }^{\prime }$:

$$S::=AA::=dA|\epsilon B::=CCC::=\epsilon$$

and the ${\mathcal{T}}_A$ and ${\mathcal{T}}_G$ given in 

Table 6. The action table ${\mathcal{T}}_A$ for the specification $\Xi$ in (5).

	d	e	$
0	$\epsilon \text{-}\mathrm{S}d,\mathrm{S}$	s	$\epsilon \text{-}\mathrm{S}d,\mathrm{R}(A,0,dA),\mathrm{R}(A,0,\epsilon ),\mathrm{R}(S,0,A),\mathrm{A}$
1			a
2			$\mathrm{R}(S,1,\epsilon )$
3	$\epsilon \text{-}\mathrm{S}d,\mathrm{S}$		$\epsilon \text{-}\mathrm{S}d,\mathrm{R}(A,1,A),\mathrm{R}(A,0,dA),\mathrm{R}(A,0,\epsilon )$
4			$\mathrm{R}(A,2,\epsilon )$
5			$\mathrm{R}(C,0,\epsilon ),\mathrm{R}(B,0,CC),\mathrm{R}(S,1,B)$
6			$\mathrm{R}(S,2,\epsilon )$
7			$\mathrm{R}(C,0,\epsilon ),\mathrm{R}(B,1,C)$
8			$\mathrm{R}(B,2,\epsilon )$

and

Table 7. The goto table ${\mathcal{T}}_G$ for the specification $\Xi$ in (5).

	S	A	B	C	d	e	$
0	1	2			3	5
1
2
3		4			3
4
5			6	7
6
7				8
8

respectively.

Parsing the input string $\omega =\mathtt{x}$ results in the GSS and the SPPF in Figure 21 and Figure 22, respectively.

The edges in the GSS are labeled with the indices of the SPPF nodes. All productions in the grammar are right-nullable. The short-circuiting reduce actions are applied in ${\mathbf{U}}_1$. $\mathrm{REDUCE}(A,1,A)$ and $\mathrm{REDUCE}(A,0,dA)$ are applied in state 3. The subtrees corresponding to $A\stackrel{*}{\Rrightarrow }\epsilon$ and $dA\stackrel{*}{\Rrightarrow }\epsilon$ are generated, resulting in the SPPF nodes 7 and 8. $\mathrm{REDUCE}(S,1,B)$ and $\mathrm{REDUCE}(B,0,CC)$ are applied in state 5. The subtrees corresponding to $B\stackrel{*}{\Rrightarrow }\epsilon$ and $CC\stackrel{*}{\Rrightarrow }\epsilon$ are generated, resulting in the SPPF nodes 1 and 4.

The sets of states ${\mathbf{P}}_{\mathfrak{i},0}$, the valid lookahead sets $\nu \left({\mathbf{P}}_{\mathfrak{i},0}\right)$, and the sets ${\mathbf{\Lambda }}_{\mathfrak{i}}$ and ${\mathbf{B}}_{\mathfrak{i}}$ for each position $\mathfrak{i}$ are given in 

Table 8. The sets of states ${\mathbf{P}}_{\mathfrak{i},0}$, the valid lookahead sets $\nu \left({\mathbf{P}}_{\mathfrak{i},0}\right)$, and the sets ${\mathbf{\Lambda }}_{\mathfrak{i}}$ and ${\mathbf{B}}_{\mathfrak{i}}$ for the specification $\Xi$ in (5).

$\mathfrak{i}$	${\mathbf{P}}_{\mathfrak{i},0}$	$\mathbf{\nu }\left({\mathbf{P}}_{\mathfrak{i},0}\right)$	${\mathbf{\Lambda }}_{\mathfrak{i}}$	${\mathbf{B}}_{\mathfrak{i}}$
0	0	$d,e,\$$	$d,e$	$d,e$
1	3, 5	$d,\$$	$	$

.

Two lookahead symbols d and e are matched at position 0. An $\epsilon$-shift action is performed from states 0 and 3. The resultant SPPF node is, in both cases, labeled $(d,0,0)$; thus, the edges share the index 5. For d, a shift action is performed from states 0 and 3, resulting in nodes labeled $(d,0,1)$; thus, the edges again share the index 6. For e, a shift action is performed from state 0, resulting in a node labeled $(e,0,1)$ and an edge labeled with index 3. There are infinitely many derivations for $A\stackrel{*}{\Rrightarrow }\epsilon$, since $A\stackrel{*}{\Rrightarrow }d\dots d$ and $d\Rrightarrow \epsilon$. In our example, one d corresponds to $\mathtt{x}$ and results in the SPPF node 6, while all others correspond to $\epsilon$ and result in SPPF nodes 5 and 8, and there can be infinitely many of them; hence, the loops in nodes 2 and 7.

Algorithm

The modifications required to generate the SPPF are purely additive. The algorithm listing is given in Algorithm 2. The algorithm now additionally accepts the trimmed-down grammar $G_{\epsilon }^{\prime }$ as an argument. The action processing functions shift and reduce are augmented to construct the SPPF nodes and label the edges. The shift now additionally accepts the start position of the lexeme, which is used as the start position of a leaf SPPF node. The function reduce now accepts the first edge of the path instead of just the successor vertex, because the SPPF node that labels this edge needs to be recorded as well. The nullable part of the right nullable productions is passed as well. That is then processed by the generate, which generates the missing subtrees. As in the algorithm proposed by Scott [17], the set $\mathcal{N}$ is introduced, which holds the SPPF nodes constructed at each position in the parse. Only the nodes in $\mathcal{N}$ need to be considered when searching for existing SPPF nodes. The labels of the SPPF nodes in $\mathcal{N}$ share the end position; thus, they could have been omitted [17]. Note that, in our case, the SPPF nodes resulting from $\epsilon \text{-}\mathrm{shift}$ and shift actions need to be added to $\mathcal{N}$ as well, as more than one node can be created. If the parse succeeded, the root SPPF node, the one that corresponds to the input string $\omega$, labels the edge between the start vertex $v_{◊}$ and a vertex w labeled with the final state of the handle-finding automaton. The nodes resulting from superfluously applied actions cannot be reached from the root SPPF node and can be removed.

Algorithm 2. RNGSGLR (0,1] parser for character-level CFL.
input	${\mathcal{T}}_A$ action table, ${\mathcal{T}}_G$ goto table, ${({\dot{M}}_{\mathfrak{i}})}_{\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}}$ automata at each position in the parse,
	$G_{\epsilon }^{\prime }$ trimmed-down grammar of nullable productions, $\omega$ character input string
output	accept or reject the character input string and return the constructed SPPF
let rngsglr2 ${\mathcal{T}}_A$ ${\mathcal{T}}_G$ ${({\dot{M}}_{\mathfrak{i}})}_{\mathfrak{i}\in \{0,\dots ,|\omega \left|\right\}}$ $G_{\epsilon }^{\prime }$ $\omega$ =
let rec actor w =
if $\exists l\in {\mathbf{\Lambda }}_{\mathfrak{i}},\epsilon \text{-}\mathrm{SHIFT}t\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),l)$
then shift $\mathfrak{i}$ w t
if $\exists l\in {\mathbf{\Lambda }}_{\mathfrak{i}},\mathrm{SHIFT}\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),l)$
then ${\mathcal{B}}_{\mathfrak{i}}:={\mathcal{B}}_{\mathfrak{i}}\cup \{l\mapsto w\}$
for each $\exists l\in {\mathbf{\Lambda }}_{\mathfrak{i}},\mathrm{REDUCE}(A,0,\eta )\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),l)$ do
reduce $(w,w)$ A 0 $\eta$
done
and reduce_actor w =
for each $\exists l\in {\mathbf{\Lambda }}_{\mathfrak{i}},\mathrm{REDUCE}(A,|\gamma |,\eta )\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),l)\wedge \left|\gamma \right|\ne 0$ do
for each successor node $w^{\prime }$ of w do
reduce $(w,w^{\prime })$ A $\left|\gamma \right|$ $\eta$
done
done
and shift_actor () =
for each match $(\mathfrak{k},l)\in {\mathbf{B}}_{\mathfrak{i}}$ do
for each $w\in {\mathcal{B}}_{\mathfrak{k}}\left(l\right),\mathrm{SHIFT}\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),l)$ do
shift $\mathfrak{k}$ w a
done
done
and reduce $(w,w^{\prime })$ A $\left|\gamma \right|$ $\eta$ =
let rec generate X =
if $\exists z\in \mathcal{N},\mathrm{LABEL}\left(z\right)=(X,\mathfrak{i},\mathfrak{i})$
then z
else
create node z labeled $(X,\mathfrak{i},\mathfrak{i})$
$\mathcal{N}:=\mathcal{N}\cup \left\{z\right\}$
for each $X\Rightarrow \kappa$ do
let $z_1,\dots ,z_{\left|\kappa \right|}$ = map generate over $\kappa =X_1\dots X_{\left|\kappa \right|}$ in
add children $z_1,\dots ,z_{\left|\kappa \right|}$ to z
done
z
done
in
let $z_1^{\prime },\dots ,z_{\left|\eta \right|}^{\prime }$ = map generate over $\eta =X_1\dots X_{\left|\eta \right|}$ in
for each v that can be reached via $(w,w^{\prime })$ along the path of length $\left|\gamma \right|$ and the corresponding $z_1\dots ,z_{\left|\gamma \right|}$ do
if $\nexists z\in \mathcal{N},\mathrm{LABEL}\left(z\right)=(A,\mathfrak{k},\mathfrak{i})\wedge \mathrm{LABEL}\left(z_1\right)=(B,\mathfrak{k},\mathfrak{f})$ then
create node z labeled $(A,{\mathfrak{k}}^{\prime },\mathfrak{i})$
$\mathcal{N}:=\mathcal{N}\cup \left\{z\right\}$
add children $z_1,\dots ,z_{\left|\gamma \right|},z_1^{\prime },\dots ,z_{\left|\eta \right|}^{\prime }$ to node z
let s = ${\mathcal{T}}_G(\mathrm{STATE}\left(v\right),A)$ in
if $\nexists u\in {\mathcal{U}}_{\mathfrak{i}},\mathrm{STATE}\left(u\right)=s$ then
add vertex u to $\mathcal{V}$, labeled s, and edge $(u,v)$ to $\mathcal{E}$ labeled z
${\mathcal{U}}_{\mathfrak{i}}:={\mathcal{U}}_{\mathfrak{i}}\cup \left\{u\right\}$
if $\left|\gamma \right|\ne 0$
then reduce_actor u
else if $(u,v)\notin \mathcal{E}$ then
add edge $(u,v)$ to $\mathcal{E}$
if $\left|\gamma \right|\ne 0$ then
for each $\exists t\in {\mathbf{\Lambda }}_{\mathfrak{i}},\mathrm{REDUCE}(B,|\theta |,\kappa )\in {\mathcal{T}}_A(\mathrm{STATE}\left(u\right),t)\wedge \left|\theta \right|\ne 0$ do
reduce $(u,v)$ B $\left|\theta \right|$ $\kappa$
done
and shift $\mathfrak{k}$ $\mathfrak{l}$ w t =
if $\nexists z\in \mathcal{N},\mathrm{LABEL}\left(z\right)=(t,\mathfrak{k},\mathfrak{l})$ then
create node z labeled $(t,\mathfrak{k},\mathfrak{l})$
$\mathcal{N}:=\mathcal{N}\cup \left\{z\right\}$
let s = ${\mathcal{T}}_G(\mathrm{STATE}\left(w\right),t)$ in
if $\nexists u\in {\mathcal{U}}_{\mathfrak{l}},\mathrm{STATE}\left(u\right)=s$ then
add vertex u to $\mathcal{V}$, labeled s, and edge $(u,w)$ to $\mathcal{E}$ labeled n
${\mathcal{U}}_{\mathfrak{l}}:={\mathcal{U}}_{\mathfrak{l}}\cup \left\{u\right\}$
else if $(u,w)\notin \mathcal{E}$
then add edge $(u,w)$ to $\mathcal{E}$ labeled z
in
let parse () =
scan at position $\mathfrak{i}$ in $\omega \$$ using ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$ tracing ${\mathring{q}}_{\mathfrak{i}}\to {\mathring{q}}_{\mathfrak{j}}^{\prime }$
get matches ${\mathbf{B}}_{\mathfrak{i}}=m\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)$
get predictions ${\mathbf{\Lambda }}_{\mathfrak{i}}=\{t\mid (\mathfrak{k},t)\in p\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)\wedge \mathfrak{k}=\mathfrak{i}\}$
${\mathcal{B}}_{\mathfrak{i}}:=\varnothing ,{\mathcal{U}}_{\mathfrak{j}}:=\varnothing$
for each vertex $w\in {\mathcal{U}}_{\mathfrak{i}}$ do
reduce_actor w
done
for each vertex $w\in {\mathcal{U}}_{\mathfrak{i}}$ do
actor w
done
$\mathcal{N}:=\varnothing$
shift_actor ()
construct ${\mathring{M}}_{\mathfrak{j}}\langle {\mathring{q}}_{\mathfrak{j}}\rangle ={\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{j}}^{\prime }\rangle |{\dot{M}}_{\mathfrak{j}}$
$\mathfrak{i}:=\mathfrak{j}$
in
add the start vertex $v_{◊}$ to $\mathcal{V}$ labeled $s_{◊}$
$\mathfrak{i}:=0,{\mathcal{U}}_0:=\left\{v_{◊}\right\}$
construct ${\mathring{M}}_0\langle {\mathring{q}}_0\rangle ={\dot{M}}_0$
while $\mathfrak{i}\le \left|\omega \right|\wedge {\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle \ne ⌀$ do
parse ()
done
if $\exists w\in {\mathcal{U}}_{\left|\omega \right|},\mathrm{ACCEPT}\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),\$)$ then
the node z labeling $(w,v_{◊})$ is the root of the forest
remove nodes not reachable from z
accept
else reject

6. Scanner Lookahead

By default, our architecture does not require a buffer. All characters are processed one by one. While all matched lookahead symbols result in a shift action due to the contextual information passed to the scanner by the parser (when using the canonical LR or an equivalent handle-finding automaton construction algorithm), some are still performed superfluously. In the example for specification $\Xi$ in (4), the shift actions are performed for the symbol c in ${\mathbf{U}}_1$, ${\mathbf{U}}_2$, and ${\mathbf{U}}_3$; however, all but the last are superfluous. The issue is that $ could have potentially followed at positions 2 or 3, and there was no way to tell if that were the case, since only one character was processed at a time. In practice, this arises when parsing embedded sub-languages. A simple example is the following specification:

$$S::=aba{r}_a="r_b={(\mathbf{\Omega }-")}^{*}$$

When parsing the input string $\omega =“\mathtt{x}\mathtt{y}\mathtt{z}”$, the symbol b will be matched at positions 1, 2, 3, because there is no way to tell whether the next character is ".

A modest extension to our architecture, which solves this problem, is the addition of the scanner lookahead [2]. Usually, when a multitude of matches is found, all except the last result in a superfluous shift action. Therefore, the use of the scanner lookahead is functionally similar to the longest match disambiguation rule. However, the longest match in certain circumstances results in an incorrect parse. As identified by Nawrocki [9], there is the longest match conflict between not necessarily distinct $t,l,l^{\prime }\in {\mathbf{T}}_{\$}$, where $\gamma ,\eta \in {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}$ and $\sigma \in {\mathbf{\Omega }}^{+}$ and $\rho ,\tau ,\pi \in {\mathbf{\Omega }}^{*}$, when:

$$S\stackrel{*}{\Rightarrow }\gamma t\eta S\stackrel{*}{\Rightarrow }\gamma l{l}^{\prime }\eta \rho \sigma \tau \in \mathcal{L}\left(r_t\right)\rho \in \mathcal{L}\left(r_l\right)\sigma \pi \in \mathcal{L}\left(r_{l^{\prime }}\right)$$

(6)

When parsing $\rho \sigma \pi$, if the longest match is used, $\rho \sigma$ is recognized by $M_t$, and then an error is raised because $\pi$ does not match $\tau$ [9]. To detect if the longest match conflict can occur, it is enough to consider only the first character of $\sigma$, as the parse is bound to be incorrect as soon as $M_t$ scans a single character past $\rho$ without returning a match. There is a special case, where $\gamma ,\eta \in {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}$ and $\sigma \in {\mathbf{\Omega }}^{+}$ and $\rho ,\pi \in {\mathbf{\Omega }}^{*}$ and $i,j\ge 0$:

$$S\stackrel{*}{\Rightarrow }\gamma t{l}^{\prime }\eta \rho {\sigma }^i\in \mathcal{L}\left(r_t\right){\sigma }^j\pi \in \mathcal{L}\left(r_{l^{\prime }}\right)$$

(7)

When parsing $\rho {\sigma }^k\pi =\rho {\sigma }^i{\sigma }^j\pi$, all possible interpretations, such as $i+j=k$, need to be considered. When using the longest match, the only possible interpretation is $i=k,j=0$. The longest match results in an incorrect parse if $M_{l^{\prime }}$ can find a match after $M_t$ enters a final state.

In the case of our architecture, it occurs if ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$ can find a match as ${\mathring{M}}_{\mathfrak{h}}\langle {\mathring{q}}_{\mathfrak{i}}^{\prime }\rangle$ has entered a final state.

• When using our architecture, this is, by default, always assumed to be true. Every matched symbol is passed to the parser.
• When using the longest match, this is always assumed to be false. The scanning continues with ${\mathring{M}}_{\mathfrak{h}}\langle {\mathring{q}}_{\mathfrak{i}}^{\prime }\rangle$ until it is about to enter a state from which no more matches can be made on the next transition.
• When using the scanner lookahead, a test is performed if that is the case. We could scan ahead using the automaton ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$ to see if it enters a state from which no more matches can be made and then continue scanning in this case. Since only continuing with the scanning results in an incorrect parse, we can always stop early and return the match to the parser. We present a variant that is equivalent to scanning a single character. We use a simple set membership test instead of scanning ahead. The match is passed to the parser only if the rest of the input string contains $a\in {\mathbf{\Omega }}_{\$}$, such that $a\in \mathrm{FIRST}\left({\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle \right)$. This is equivalent to one character of lookahead in a character-level parser.

To determine the lookahead characters automatically using $G^{\prime }$ we introduce a valid-lookahead-lookahead multi-map, $\xi :\mathbf{S}\times {\mathbf{T}}_{\$}\to \mathcal{P}\left({\mathbf{T}}_{\$}\right)$, which is the set of lookahead symbols for each lookahead symbol in the valid lookahead set. We identified the following approaches to compute $\xi$:

• The simplest involves using the lookahead sets in a handle-finding automaton constructed using two symbols of lookahead as proposed by Keynes [8]. For each state $s\in \mathbf{S}$, the shift items $(A::=\gamma •a\eta tl)\in s$ are converted into $\{(s,t^{\prime })\mapsto l^{\prime }\mid t^{\prime }{l}^{\prime }\in {\mathrm{FIRST}}_2\left(a\eta tl\right)\}$, and the reduce items $(A::=\beta •tl)\in s$ are converted to $\left\{\right(s,t)\mapsto l\}$. The issue with this approach is that construction of automata using two symbols of the lookahead is inefficient, as also noted by Keynes [8]. Additionally, it computes $\xi$ indirectly. It needs to compute two tokens of lookahead per item, which we have no use for, and then collapse it. We did not manage to make this approach reasonably efficient for large grammars.
• An approach based on exploring the right context of each LALR(1) handle-finding automaton state manually is given by Nawrocki [9].
• We found an efficient approach that allows us to compute $\xi$ directly. We construct the LALR(1) handle-finding automaton by converting the LR(0) handle-finding automaton to SLR [1]. The method works by constructing a grammar where each symbol is enhanced with the state of the LR(0) handle-finding automaton. The lookahead is computed as the $\mathrm{FOLLOW}$ sets of the enhanced non-terminal symbols. The $\xi$ is computed as the $\mathrm{FOLLOW}$ sets of the enhanced terminal symbols. In a sense, the $\xi$ corresponds to the lookahead that would be used when performing the reduce action for the symbol $t\in \mathbf{T}$. The downside of the method is that it is tied closely to the handle-finding automaton construction method.

The scanner lookahead multi-map ${\xi }^{\prime }:\mathbf{S}\times {\mathbf{T}}_{\$}\to \mathcal{P}\left({\mathbf{\Omega }}_{\$}\right)$ is then computed as

$${\xi }^{\prime }(s,t)=\bigcup _{l\in \xi (s,t)}\mathrm{FIRST}\left(M_l\right)$$

For a set of handle-finding automaton states, ${\xi }^{\prime }:\Pi \times {\mathbf{T}}_{\$}\to \mathcal{P}\left({\mathbf{\Omega }}_{\$}\right)$ is defined as

$${\xi }^{\prime }(\mathbf{P},t)=\bigcup _{s\in \mathbf{P}}{\xi }^{\prime }(s,t)$$

The $m:{\mathring{\mathbf{Q}}}_{\mathfrak{i}}\times {\mathbf{\Omega }}_{\$}\to \mathcal{P}\left(\right\{0,\dots ,\left|\omega \right|\}\times {\mathbf{T}}_{\$})$ and $p:{\mathring{\mathbf{Q}}}_{\mathfrak{i}}\times {\mathbf{\Omega }}_{\$}\to \mathcal{P}\left(\right\{0,\dots ,\left|\omega \right|\}\times {\mathbf{T}}_{\$})$ are redefined as:

$$\begin{array}{cc}\hfill m({\mathring{q}}_{\mathfrak{j}}^{\prime },a)& =\bigcup _{\mathfrak{k}\in \{0,\dots ,\mathfrak{j}\}}\left\{(\mathfrak{k},t)\mid t\in m({\mathbf{P}}_{\mathfrak{k},0},{\dot{q}}_{\mathfrak{k}/\mathfrak{j}})\wedge a\in {\xi }^{\prime }({\mathbf{P}}_{\mathfrak{k},0},t)\right\}\hfill \\ \hfill p({\mathring{q}}_{\mathfrak{j}}^{\prime },a)& =\bigcup _{\mathfrak{k}\in \{0,\dots ,\mathfrak{j}\}}\left\{(\mathfrak{k},t)\mid t\in p({\mathbf{P}}_{\mathfrak{k},0},{\dot{q}}_{\mathfrak{k}/\mathfrak{j}})\wedge a\in {\xi }^{\prime }({\mathbf{P}}_{\mathfrak{k},0},t)\right\}\hfill \end{array}$$

The scanning process is then modified as follows: The input string $\omega$ is extended with $\$\$$. At each position $\mathfrak{i}$ in the parse a string ${\psi }_{(\mathfrak{i},\mathfrak{j})}\in {\mathbf{\Omega }}_{\$}^{+}$ is recognized in the rest of the input string ${\omega }^{\prime }$ using ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$ as usual, such that ${\omega }^{\prime }={\psi }_{(\mathfrak{i},\mathfrak{j})}{\omega }^{″}$. Then we peek at the next character a, such that ${\omega }^{″}=a{\omega }^{‴}$. If none of the matched symbols pass the test, then $m({\mathring{q}}_{\mathfrak{j}}^{\prime },a)=\varnothing$, and as a result, ${\dot{M}}_{\mathfrak{j}}=⌀$; therefore, the scanning immediately continues with ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{j}}^{\prime }\rangle$. Otherwise, $m({\mathring{q}}_{\mathfrak{j}}^{\prime },a)$ and $p({\mathring{q}}_{\mathfrak{j}}^{\prime },a)$ are passed to the parser and the scanning continues with ${\mathring{M}}_{\mathfrak{j}}\langle {\mathring{q}}_{\mathfrak{j}}\rangle$.

7. Proof of Correctness

A recognizer is correct for a given specification $\Xi$ if it accepts an input string $\omega \in {\mathbf{\Omega }}^{*}$, if and only if $\omega \in \mathcal{L}(\Xi )$, and otherwise rejects it. First, we will prove if our architecture accepts $\omega$, then $\omega \in \mathcal{L}(\Xi )$.

Lemma 1.

For each vertex $u\in {\mathbf{U}}_{\mathfrak{i}}$, where $(A::=\gamma X•\eta l)\in \mathrm{STATE}\left(u\right)$, there must also exist a vertex $w\in {\mathbf{U}}_{\mathfrak{h}}$, such that $(A::=\gamma •X\eta l)\in \mathrm{STATE}\left(w\right)$, where $\gamma ,\eta \in {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}$, and there must exist an edge $w\stackrel{X}{\leftarrow }u$ in the GSS.

Proof. 

This follows trivially from the definition of the handle-finding automaton and the basic operation of the RNGLR parser [15]. □

Lemma 2.

For each edge $w\stackrel{t}{\leftarrow }u$, such that $w\in {\mathbf{U}}_{\mathfrak{h}}$ and $u\in {\mathbf{U}}_{\mathfrak{i}}$, created using a shift action, ${\psi }_{(\mathfrak{h},\mathfrak{i})}\in \mathcal{L}\left(t\right)$.

Proof. 

The shift action is performed if $(\mathfrak{h},t)\in m\left({\mathring{q}}_{\mathfrak{i}}^{\prime }\right)$ after ${\mathring{M}}_{\mathfrak{h}}\langle {\mathring{q}}_{\mathfrak{h}}\rangle$ traces a path ${\mathring{q}}_{\mathfrak{h}}\stackrel{{\psi }_{(\mathfrak{h},\mathfrak{i})}}{\to }{\mathring{q}}_{\mathfrak{i}}^{\prime }$. That means the match was found by the automaton ${\dot{M}}_{\mathfrak{h}}$, such that ${\dot{q}}_{\mathfrak{h}/\mathfrak{h}}\stackrel{{\psi }_{(\mathfrak{h},\mathfrak{i})}}{\to }{\dot{q}}_{\mathfrak{h}/\mathfrak{i}}$, and that $t\in m({\mathbf{P}}_{\mathfrak{h},0},{\dot{q}}_{\mathfrak{h}/\mathfrak{i}})$. Since $m({\mathbf{P}}_{\mathfrak{h},0},{\dot{q}}_{\mathfrak{h}/\mathfrak{i}})=\nu \left({\mathbf{P}}_{\mathfrak{h},0}\right)\cap m\left({\dot{q}}_{\mathfrak{h}/\mathfrak{i}}\right)$, which means that $t\in \nu \left({\mathbf{P}}_{\mathfrak{h},0}\right)$ and $t\in m\left({\dot{q}}_{\mathfrak{h}/\mathfrak{i}}\right)$. The automaton ${\dot{M}}_{{\mathbf{T}}_{\$}}$ is a union automaton, which means it finds a match if any of the constitutive automata find a match. The only automaton where $t\in m\left(q_t\right)$ is $M_t$, which means it must necessarily be the one that recognizes ${\psi }_{(\mathfrak{h},\mathfrak{i})}$. Therefore, ${\psi }_{(\mathfrak{h},\mathfrak{i})}\in \mathcal{L}\left(M_t\right)$, and by definition of $M_t$, ${\psi }_{(\mathfrak{h},\mathfrak{i})}\in \mathcal{L}\left(t\right)$. □

Lemma 3.

For each edge $w\stackrel{X}{\leftarrow }u$, such that $w\in {\mathbf{U}}_{\mathfrak{i}}$ and $u\in {\mathbf{U}}_{\mathfrak{i}}$, created using an ε-shift action or an ε-reduce action, ${\psi }_{(\mathfrak{i},\mathfrak{i})}\in \mathcal{L}\left(X\right)$.

Proof. 

An $\epsilon$-shift action or $\epsilon$-reduce action is only performed if $X\stackrel{*}{\Rrightarrow }\epsilon$. Since ${\psi }_{(\mathfrak{i},\mathfrak{i})}=\epsilon$ at any position, ${\psi }_{(\mathfrak{i},\mathfrak{i})}\in \mathcal{L}\left(X\right)$. □

Lemma 4.

For each edge $w\stackrel{A}{\leftarrow }u$, such that $w\in {\mathbf{U}}_{\mathfrak{h}}$ and $u\in {\mathbf{U}}_{\mathfrak{i}}$, created using a reduce action, where $\left|\gamma \right|>0$, ${\psi }_{(\mathfrak{h},\mathfrak{i})}\in \mathcal{L}\left(A\right)$.

Proof. 

We will prove the statement by induction. If the GSS does not contain any edges, the statement is trivially satisfied. If the statement holds for all of the edges in the GSS, then it must hold after $w\stackrel{A}{\leftarrow }u$ has been added. The reduce action $\mathrm{REDUCE}(A,|\gamma |,\eta )\in {\mathcal{T}}_A(\mathrm{STATE}\left(v\right),l)$, where $v\in {\mathbf{U}}_{\mathfrak{i}}$, is performed if there exists a path $v_0\stackrel{X_1}{\leftarrow }\dots \stackrel{X_{\left|\gamma \right|}}{\leftarrow }{v}_{\left|\gamma \right|}$, where $\gamma =X_1\dots X_{\left|\gamma \right|}$ and $v_0=w$ and $v_{\left|\gamma \right|}=v$. For each $X_{i\in \{1,\dots ,|\gamma \left|\right\}}$, where $v_{i-1}\in {\mathbf{U}}_{\mathfrak{e}}$ and $v_i\in {\mathbf{U}}_{\mathfrak{f}}$, ${\psi }_{(\mathfrak{e},\mathfrak{f})}\in \mathcal{L}\left(X_i\right)$. This statement holds for the edge $v_{i-1}\stackrel{X_i}{\leftarrow }{v}_i$:

• Due to Lemma 2, if it was created as a result of a shift action;
• Due to Lemma 3, if it was created as a result of an $\epsilon$-shift or $\epsilon$-reduce action; or
• Due to the induction hypothesis, if it was created as a result of a reduce action.

This exhausts all possible ways an edge can be created. The reduce action entry was added to the actions table because $(A::=X_1\dots X_{\left|\gamma \right|}•\eta l)\in \mathrm{STATE}\left(v\right)$. That means $A\stackrel{*}{\Rightarrow }{X}_1\dots X_{\left|\gamma \right|}\eta$, where $\eta \stackrel{*}{\Rrightarrow }\epsilon$, and as a result ${\psi }_{(\mathfrak{h},\mathfrak{i})}\in \mathcal{L}\left(A\right)$. □

Theorem 1.

For each edge $w\stackrel{X}{\leftarrow }u$, such that $w\in {\mathbf{U}}_{\mathfrak{h}}$ and $u\in {\mathbf{U}}_{\mathfrak{i}}$, in the GSS, ${\psi }_{(\mathfrak{h},\mathfrak{i})}\in \mathcal{L}\left(X\right)$.

Proof. 

An edge can only be created as a result of a shift, an $\epsilon$-shift, an $\epsilon$-reduce or a reduce action, where $\left|\gamma \right|>0$; therefore, the statement holds due to Lemmas 2, 3, 4. □

Theorem 2.

If our architecture accepts ω then $\omega \in \mathcal{L}(\Xi )$.

Proof. 

If our architecture accepts $\omega$, it means there must exist a vertex $w\in {\mathbf{U}}_{\left|\omega \right|}$, such that $\mathrm{ACCEPT}\in {\mathcal{T}}_A(\mathrm{STATE}\left(w\right),\$)$. This implies $(S^{\prime }::=S•\$)\in \mathrm{STATE}\left(w\right)$. By Lemma 1, there must exist a vertex v, such that $(S^{\prime }::=•S\$)\in \mathrm{STATE}\left(v\right)$ and an edge $v\stackrel{S}{\leftarrow }w$. By the definition of the handle-finding automaton, the only such vertex is $v_{◊}$. Therefore, there exists an edge $v_{◊}\leftarrow w$, such that $v\in {\mathbf{U}}_0$ and $w\in {\mathbf{U}}_{\left|\omega \right|}$. By Theorem 1, $\omega \in \mathcal{L}\left(S\right)$, therefore, $\omega \in \mathcal{L}(\Xi )$ [15]. □

Second, we will prove that if $\omega \in \mathcal{L}(\Xi )$, then our architecture accepts $\omega \in {\mathbf{\Omega }}^{*}$. We will do so by showing that our architecture accepts $\omega$ exactly when a character-level RNGLR parser accepts $\omega$ using $\widehat{G}$. The correctness of RNGLR was proven by Scott [17].

We will formulate the lookahead as the right context. The item right context is the grammar that describes the language of all strings that can follow an item in state $s\in \mathbf{S}$. For an item of the form $(A::=•\beta )\in s$, which was added as a result of the $\epsilon$-closure on an item of the form $(B::=\gamma •A{\eta }^{\prime })\in s$, a production $F_{s,(A::=•\beta )}::={\eta }^{\prime }{F}_{s,(B::=\gamma •A{\eta }^{\prime })}$ is added to the grammar. Shifting over a symbol does not change the item right context; thus, for the items of the form $(B::=\gamma •X{\eta }^{\prime })\in s^{\prime }$ and $(B::=\gamma X•{\eta }^{\prime })\in s$, a production of the form is added $F_{s^{\prime },(B::=\gamma •X{\eta }^{\prime })}::=F_{s,(B::=\gamma X•{\eta }^{\prime })}$. The dot right context is grammar that describes the language of all strings that can follow the dot. The grammar is initially constructed in the same way as for the item’s right context. Then, for each item of the form $(B::=\gamma •X{\eta }^{\prime })\in s$, a production $D_{s,(B::=\gamma •X{\eta }^{\prime })}::=X{\eta }^{\prime }{F}_{s,(B::=\gamma •X{\eta }^{\prime })}$ is added [1]. We assume the character-level RNGLR uses the whole right context as the lookahead; in that way, it cannot perform superfluous actions. Our architecture performs strictly worse in that regard.

To avoid confusion, the symbols that belong to RNGLR will be annotated with $\left(1\right)$ and those that belong to our architecture will be annotated with $\left(2\right)$.

Lemma 5.

For all states $s^{\left(1\right)}\in {\mathbf{S}}^{\left(1\right)}$ and $s^{\left(2\right)}\in {\mathbf{S}}^{\left(2\right)}$, if $B\in {\mathbf{N}}^{\prime }$, where $\gamma ,\eta \in {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}$:

$$(B::=\gamma •\eta )\in s^{\left(1\right)}\iff (B::=\gamma •\eta )\in s^{\left(2\right)}$$

Proof. 

We will prove the statement by induction. The base cases are the states $s_{◊}^{\left(1\right)}\in {\mathbf{S}}^{\left(1\right)}$ and $s_{◊}^{\left(2\right)}\in {\mathbf{S}}^{\left(2\right)}$. We will prove the statement holds for the base case, again by induction. Initially, $(S^{\prime }::=•S)\in s_{◊}^{\left(1\right)}$ and $(S^{\prime }::=•S)\in s_{◊}^{\left(2\right)}$. If we assume $S^{\prime }\in {\mathbf{N}}^{\prime }$ the statement holds trivially as the initial items exist for both $\widehat{G}$ and $G^{\prime }$ by the definition of the handle finding automaton. We will prove that, as the items are expanded using $\epsilon$-closure, the statement still holds: By the induction hypothesis, there exist items $(B::=•X{\eta }^{\prime })\in s_{◊}^{\left(1\right)}$ and $(B::=•X{\eta }^{\prime })\in s_{◊}^{\left(2\right)}$, where $\eta =X{\eta }^{\prime }$ and $\eta \in {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}$, or there are no such items in $s_{◊}^{\left(1\right)}$ and $s_{◊}^{\left(2\right)}$. If $X\in {\mathbf{N}}^{\prime }$, the items $(X::=•\beta )$ are added for both $\widehat{G}$ and $G^{\prime }$. That is because productions $X::=\beta$, such that $X\in {\mathbf{N}}^{\prime }$, exist in both grammars, by the definition of $\widehat{G}$. Moreover $\beta \in {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}$; therefore, the statement continues to hold.

Therefore, the statement holds for the base case. We will prove that the statement still holds after new states $s^{{}^{\prime }\left(1\right)}\in {\mathbf{S}}^{\left(1\right)}$ and $s^{{}^{\prime }\left(2\right)}\in {\mathbf{S}}^{\left(2\right)}$ are added. Each state is created from an existing state $s^{\left(1\right)}\in {\mathbf{S}}^{\left(1\right)}$ and $s^{\left(2\right)}\in {\mathbf{S}}^{\left(2\right)}$. By the induction hypothesis, there exist items $(B::=\gamma •X{\eta }^{\prime })\in s^{\left(1\right)}$ and $(B::=\gamma •X{\eta }^{\prime })\in s^{\left(2\right)}$, where $\eta =X{\eta }^{\prime }$ and $\gamma ,\eta \in {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}$, or there are no such items in $s_{◊}^{\left(1\right)}$ and $s_{◊}^{\left(2\right)}$. If they exist, then a new state $s^{{}^{\prime }\left(1\right)}$ and an edge $s^{\left(1\right)}\stackrel{X}{\to }{s}^{{}^{\prime }\left(1\right)}$ are created for $\widehat{G}$ and new states $s^{{}^{\prime }\left(2\right)}$ and $s^{\left(2\right)}\stackrel{X}{\to }{s}^{{}^{\prime }\left(2\right)}$ are created for $G^{\prime }$. Initially, there exist items $(B::=\gamma X•{\eta }^{\prime })\in s^{{}^{\prime }\left(1\right)}$ and $(B::=\gamma X•{\eta }^{\prime })\in s^{{}^{\prime }\left(2\right)}$. if ${\eta }^{\prime }=\epsilon$, no more items are added and the statement continues to hold. Otherwise, if ${\eta }^{\prime }=X^{\prime }{\eta }^{″}$, then the items are expanded using $\epsilon$-closure. We can use the same reasoning for $s_{◊}^{\left(1\right)}$ and $s_{◊}^{\left(2\right)}$, to show that the statement still holds for the created items. Therefore, the statement holds for all $s^{\left(1\right)}\in {\mathbf{S}}^{\left(1\right)}$ and $s^{\left(2\right)}\in {\mathbf{S}}^{\left(2\right)}$. □

Corollary 1.

For all states $s^{\left(1\right)},s^{{}^{\prime }\left(1\right)}\in {\mathbf{S}}^{\left(1\right)}$ and $s^{\left(2\right)},s^{{}^{\prime }\left(2\right)}\in {\mathbf{S}}^{\left(2\right)}$, when $B\in {\mathbf{N}}^{\prime }$, the item right context and the dot right context coincide for both $\widehat{G}$ and $G^{\prime }$:

$$\begin{array}{ccccccc}\hfill F_{s^{\left(1\right)},(A::=•\beta )}& ::=\hfill & {\eta }^{\prime }{F}_{s^{\left(1\right)},(B::=\gamma •A{\eta }^{\prime })}\hfill & ⟺& \hfill F_{s^{\left(2\right)},(A::=•\beta )}& ::=\hfill & {\eta }^{\prime }{F}_{s^{\left(2\right)},(B::=\gamma •A{\eta }^{\prime })}\hfill \\ \hfill F_{s^{{}^{\prime }\left(1\right)},(B::=\gamma •X{\eta }^{\prime })}& ::=\hfill & F_{s^{\left(1\right)},(B::=\gamma X•{\eta }^{\prime })}\hfill & ⟺& \hfill F_{s^{{}^{\prime }\left(2\right)},(B::=\gamma •X{\eta }^{\prime })}& ::=\hfill & F_{s^{\left(2\right)},(B::=\gamma X•{\eta }^{\prime })}\hfill \\ \hfill D_{s^{\left(1\right)},(B::=\gamma •X{\eta }^{\prime })}& ::=\hfill & X{\eta }^{\prime }{F}_{s^{\left(1\right)},(B::=\gamma •X{\eta }^{\prime })}\hfill & ⟺& \hfill D_{s^{\left(2\right)},(B::=\gamma •X{\eta }^{\prime })}& ::=\hfill & X{\eta }^{\prime }{F}_{s^{\left(2\right)},(B::=\gamma •X{\eta }^{\prime })}\hfill \end{array}$$

This follows trivially from Lemma 5.

Lemma 6.

${\Gamma }^{\left(2\right)}$ contains all paths labeled $\iota \in {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}$ in ${\Gamma }^{\left(1\right)}$, for all derivations $S\stackrel{*}{\Rightarrow }\iota {\omega }^{\prime }$.

Proof. 

The $\iota$ are the configurations of the stack in LR parsing [1]. We will prove the statement by induction. The base cases are the empty paths starting at $v_{◊}^{\left(1\right)}$ and $v_{◊}^{\left(2\right)}$, which exist by the definition of both recognizers. By the induction hypothesis, there are paths:

$$v_{◊}^{\left(1\right)}\stackrel{\iota }{\leftarrow }{w}^{\left(1\right)}{v}_{◊}^{\left(2\right)}\stackrel{\iota }{\leftarrow }{w}^{\left(2\right)}$$

We will prove that if an edge $w^{\left(1\right)}\stackrel{X}{\leftarrow }{u}^{\left(1\right)}$, where $X\in ({\mathbf{N}}^{\prime }\cup \mathbf{T})$, is added to ${\Gamma }^{\left(1\right)}$, an edge $w^{\left(2\right)}\stackrel{X}{\leftarrow }{u}^{\left(2\right)}$ is added to ${\Gamma }^{\left(2\right)}$. In either algorithm, an edge can only be added by processing an action. We will show that, if an action is performed by a character-level RNGLR, it is also performed by our architecture.

1.

A reduce action $\mathrm{REDUCE}(A,|\gamma |,\eta )\in {\mathcal{T}}_A^{\left(1\right)}(\mathrm{STATE}\left(v^{\left(1\right)}\right),{\omega }^{\prime })$ for production $A::=\gamma \eta$, such that $A\in {\mathbf{N}}^{\prime }$ and $\left|\gamma \right|>0$ and $\eta \stackrel{*}{\Rightarrow }\epsilon$, is performed using a character-level RNGLR if there exists a path $v_{◊}^{\left(1\right)}\stackrel{\alpha }{\leftarrow }{w}^{\left(1\right)}\stackrel{\gamma }{\leftarrow }{v}^{\left(1\right)}$, where $w^{\left(1\right)}\in {\mathbf{U}}_{\mathfrak{h}}^{\left(1\right)}$ and $v^{\left(1\right)}\in {\mathbf{U}}_{\mathfrak{i}}^{\left(1\right)}$, such that $(B::=\sigma •A\tau )\in \mathrm{STATE}\left(w^{\left(1\right)}\right)$ and $(A::=\gamma •\eta )\in \mathrm{STATE}\left(v^{\left(1\right)}\right)$, and if the rest of the input matches ${\omega }^{\prime }$, such that $F_{\mathrm{STATE}\left(v^{\left(1\right)}\right),(A::=\gamma •\eta )}\stackrel{*}{\Rightarrow }{\omega }^{\prime }$.

By the induction hypothesis, there exists a path $v_{◊}^{\left(2\right)}\stackrel{\alpha }{\leftarrow }{w}^{\left(2\right)}\stackrel{\gamma }{\leftarrow }{v}^{\left(2\right)}$, where $w^{\left(2\right)}\in {\mathbf{U}}_{\mathfrak{h}}^{\left(2\right)}$ and $v^{\left(2\right)}\in {\mathbf{U}}_{\mathfrak{i}}^{\left(2\right)}$, as well. By Lemma 5, $(B::=\sigma •A\tau )\in \mathrm{STATE}\left(w^{\left(2\right)}\right)$ and $(A::=\gamma •\eta )\in \mathrm{STATE}\left(v^{\left(2\right)}\right)$, therefore $\mathrm{REDUCE}(A,|\gamma |,\eta )\in {\mathcal{T}}_A^{\left(2\right)}(\mathrm{STATE}\left(v^{\left(2\right)}\right),l)$. That means $l\in \nu \left(\mathrm{STATE}\left(v^{\left(2\right)}\right)\right)$. The reduce action is performed if $l\in p\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)$ after ${\mathring{M}}_{\mathfrak{i}}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$ traces a path ${\mathring{q}}_{\mathfrak{i}}\stackrel{{\psi }_{(\mathfrak{i},\mathfrak{j})}}{\to }{\mathring{q}}_{\mathfrak{j}}^{\prime }$. That will happen if $l\Rrightarrow {\psi }_{(\mathfrak{i},\mathfrak{j})}{\psi }^{\prime }$. Since $F_{\mathrm{STATE}\left(v^{\left(1\right)}\right),(A::=\gamma •\eta )}\stackrel{*}{\Rightarrow }\kappa \stackrel{*}{\Rightarrow }{\omega }^{\prime }$, then by Corollary 1, $F_{\mathrm{STATE}\left(v^{\left(2\right)}\right),(A::=\gamma •\eta )}\stackrel{*}{\Rightarrow }l{\kappa }^{\prime }$, where $\kappa =l{\kappa }^{\prime }$ and $\kappa ,{\kappa }^{\prime }\in {\mathbf{T}}^{*}$.

Therefore, it is guaranteed that ${\omega }^{\prime }={\psi }_{(\mathfrak{i},\mathfrak{j})}{\psi }^{\prime }{\omega }^{″}$, where $l{\kappa }^{\prime }\stackrel{*}{\Rrightarrow }{\psi }_{(\mathfrak{i},\mathfrak{j})}{\psi }^{\prime }{\omega }^{″}$. Thus, the action is also performed by our architecture. The relevant parts of the resulting ${\Gamma }^{\left(1\right)}$ and ${\Gamma }^{\left(2\right)}$ are presented in Figure 23.

Figure 23. The relevant part of ${\Gamma }^{\left(1\right)}$ after the reduce action is performed by the character-level RNGLR and the corresponding part of ${\Gamma }^{\left(2\right)}$ after the reduce action is performed by our architecture.

Figure 23. The relevant part of ${\Gamma }^{\left(1\right)}$ after the reduce action is performed by the character-level RNGLR and the corresponding part of ${\Gamma }^{\left(2\right)}$ after the reduce action is performed by our architecture.

2.

The conditions for performing an $\epsilon \text{-}\mathrm{reduce}$ action by the character-level RNGLR and our architecture are the same as in 1, except in this case, $v^{\left(1\right)}=w^{\left(1\right)}$ and $v^{\left(2\right)}=w^{\left(2\right)}$, as $\left|\gamma \right|=0$. The relevant parts of the resulting ${\Gamma }^{\left(1\right)}$ and ${\Gamma }^{\left(2\right)}$ are presented in Figure 24.

Figure 24. The relevant part of ${\Gamma }^{\left(1\right)}$ after the $\epsilon$-reduce action is performed by the character-level RNGLR and the corresponding part of ${\Gamma }^{\left(2\right)}$ after the $\epsilon$-reduce action is performed by our architecture.

Figure 24. The relevant part of ${\Gamma }^{\left(1\right)}$ after the $\epsilon$-reduce action is performed by the character-level RNGLR and the corresponding part of ${\Gamma }^{\left(2\right)}$ after the $\epsilon$-reduce action is performed by our architecture.

3.

The conditions for performing a reduce action $\mathrm{REDUCE}(t,|\gamma |,\eta )\in {\mathcal{T}}_A^{\left(1\right)}(\mathrm{STATE}\left(v^{\left(1\right)}\right),{\omega }^{\prime })$ for production $t::=\gamma \eta$, such that $\left|\gamma \right|>0$ and $\eta \stackrel{*}{\Rightarrow }\epsilon$ and $t\in \mathbf{T}$, are the same as in 1.

By the induction hypothesis, there exists a path $v_{◊}^{\left(2\right)}\stackrel{\alpha }{\leftarrow }{w}^{\left(2\right)}$, where $w^{\left(2\right)}\in {\mathbf{U}}_{\mathfrak{h}}^{\left(2\right)}$. The path does not exist for $\gamma$, because $t\notin {\mathbf{N}}^{\prime }$; therefore, $\gamma \notin {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}$. By Lemma 5, $(B::=\sigma •t\tau )\in \mathrm{STATE}\left(w^{\left(2\right)}\right)$. Since t is a terminal symbol in $G^{\prime }$, there exists a shift action $\mathrm{SHIFT}\in {\mathcal{T}}_A^{\left(2\right)}(\mathrm{STATE}\left(w^{\left(2\right)}\right),t)$. This means $t\in \nu \left(\mathrm{STATE}\left(w^{\left(2\right)}\right)\right)$. The shift action is performed if $(\mathfrak{h},t)\in m\left({\mathring{q}}_{\mathfrak{i}}^{\prime }\right)$ after $\mathring{M}\langle {\mathring{q}}_{\mathfrak{h}}\rangle$ traces a path ${\mathring{q}}_{\mathfrak{h}}\stackrel{{\psi }_{(\mathfrak{h},\mathfrak{i})}}{\to }{\mathring{q}}_{\mathfrak{i}}^{\prime }$. That will happen if $t\Rrightarrow {\psi }_{(\mathfrak{h},\mathfrak{i})}$. The reduce action creates an edge $w^{\left(1\right)}\stackrel{X}{\leftarrow }{u}^{\left(1\right)}$, such that $w^{\left(1\right)}\in {\mathbf{U}}_{\mathfrak{h}}^{\left(1\right)}$ and $u^{\left(1\right)}\in {\mathbf{U}}_{\mathfrak{i}}^{\left(1\right)}$, in ${\Gamma }^{\left(1\right)}$, which means $t\stackrel{*}{\Rightarrow }{\psi }_{(\mathfrak{h},\mathfrak{i})}$ [15,17]. As a result, a match will necessarily be found for t and the action will also be performed by our architecture. Since $D_{\mathrm{STATE}\left(w^{\left(1\right)}\right),(B::=\sigma •t\tau )}\stackrel{*}{\Rightarrow }t\kappa \stackrel{*}{\Rightarrow }{\psi }_{(\mathfrak{h},\mathfrak{i})}{\omega }^{\prime }$, then by Corollary 1, $D_{\mathrm{STATE}\left(w^{\left(2\right)}\right),(B::=\sigma •t\tau )}\stackrel{*}{\Rightarrow }tl{\kappa }^{\prime }$, where $t\kappa =tl{\kappa }^{\prime }$ and $\kappa ,{\kappa }^{\prime }\in {\mathbf{T}}^{*}$. Furthermore, ${\psi }_{(\mathfrak{h},\mathfrak{i})}{\omega }^{\prime }={\psi }_{(\mathfrak{h},\mathfrak{i})}a{\psi }^{″}{\omega }^{″}$, where $l\Rrightarrow a{\psi }^{″}$. That means $l\in \xi (\mathrm{STATE}\left(w^{\left(2\right)}\right),t)$ and $a\in {\xi }^{\prime }(\mathrm{STATE}\left(w^{\left(2\right)}\right),t)$, and it is guaranteed that a will be found in the rest of the input string. Therefore, the action will also be performed if the scanner lookahead is used. The relevant part of the resulting ${\Gamma }^{\left(1\right)}$ and ${\Gamma }^{\left(2\right)}$ are presented in Figure 25.

Figure 25. The relevant part of ${\Gamma }^{\left(1\right)}$ after the reduce action is performed by the character-level RNGLR and the corresponding part of ${\Gamma }^{\left(2\right)}$ after the shift action is performed by our architecture.

Figure 25. The relevant part of ${\Gamma }^{\left(1\right)}$ after the reduce action is performed by the character-level RNGLR and the corresponding part of ${\Gamma }^{\left(2\right)}$ after the shift action is performed by our architecture.

4.

The conditions for performing an $\epsilon$-reduce action $\mathrm{REDUCE}(t,|\gamma |,\eta )\in {\mathcal{T}}_A^{\left(1\right)}(\mathrm{STATE}\left(w^{\left(1\right)}\right),{\omega }^{\prime })$, such that $\left|\gamma \right|=0$ and $\eta \stackrel{*}{\Rightarrow }\epsilon$ and $t\in \mathbf{T}$, are the same as in 1. By the induction hypothesis, there exists a path $v_{◊}^{\left(2\right)}\stackrel{\alpha }{\leftarrow }{w}^{\left(2\right)}$, where $w^{\left(2\right)}\in {\mathbf{U}}_{\mathfrak{h}}^{\left(2\right)}$. The path does not exist for $\gamma$, because $\left|\gamma \right|=0$. By Lemma 5, $(B::=\sigma •t\tau )\in \mathrm{STATE}\left(w^{\left(2\right)}\right)$. Since $\left|\gamma \right|=0$, there exists an $\epsilon$-shift action $\epsilon \text{-}\mathrm{SHIFT}t\in {\mathcal{T}}_A^{\left(2\right)}(\mathrm{STATE}\left(w^{\left(2\right)}\right),l)$. This means $l\in \nu \left(\mathrm{STATE}\left(w^{\left(2\right)}\right)\right)$. The $\epsilon$-shift action is performed if $l\in p\left({\mathring{q}}_{\mathfrak{j}}^{\prime }\right)$ after $\mathring{M}\langle {\mathring{q}}_{\mathfrak{i}}\rangle$ traces a path ${\mathring{q}}_{\mathfrak{i}}\stackrel{{\psi }_{(\mathfrak{i},\mathfrak{j})}}{\to }{\mathring{q}}_{\mathfrak{j}}^{\prime }$. That will happen if $l\Rrightarrow {\psi }_{(\mathfrak{i},\mathfrak{j})}{\psi }^{\prime }$. Since $D_{\mathrm{STATE}\left(w^{\left(1\right)}\right),(B::=\sigma •t\tau )}\stackrel{*}{\Rightarrow }t\kappa \stackrel{*}{\Rightarrow }{\omega }^{\prime }$, where $t\stackrel{*}{\Rightarrow }\epsilon$, then by Corollary 1, $D_{\mathrm{STATE}\left(w^{\left(2\right)}\right),(B::=\sigma •t\tau )}\stackrel{*}{\Rightarrow }tl{\kappa }^{\prime }$, where $t\kappa =tl{\kappa }^{\prime }$ and $t\Rrightarrow \epsilon$ and $\kappa ,{\kappa }^{\prime }\in {\mathbf{T}}^{*}$. Therefore, it is guaranteed that ${\omega }^{\prime }={\psi }_{(\mathfrak{i},\mathfrak{j})}{\psi }^{\prime }{\omega }^{″}$, where $tl{\kappa }^{\prime }\stackrel{*}{\Rrightarrow }{\psi }_{(\mathfrak{i},\mathfrak{j})}{\psi }^{\prime }{\omega }^{″}$. Thus, the action is also performed by our architecture. The relevant part of the resulting ${\Gamma }^{\left(1\right)}$ and ${\Gamma }^{\left(2\right)}$ are presented in Figure 26.

This exhausts all possible combinations of actions, where $X\in ({\mathbf{N}}^{\prime }\cup \mathbf{T})$; therefore, the statement holds for all paths $\iota$. ${\Gamma }^{\left(2\right)}$ indeed contains all paths labeled $\iota \in {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}$ in ${\Gamma }^{\left(1\right)}$. □

Theorem 3.

If $\omega \in \mathcal{L}(\Xi )$, then our architecture accepts ω.

Proof. 

The (character-level) RNGLR algorithm accepts a string $\omega$ if there exists an edge $v_{◊}^{\left(1\right)}\stackrel{S}{\leftarrow }{w}^{\left(1\right)}$ in ${\Gamma }^{\left(1\right)}$, such that $v_{◊}^{\left(1\right)}\in {\mathbf{U}}_0^{\left(1\right)}$ and $w^{\left(1\right)}\in {\mathbf{U}}_{\left|\omega \right|}^{\left(1\right)}$ [15,17]. By Lemma 6, since $S\in {\mathbf{N}}^{\prime }$ is a path of length one, it exists in ${\Gamma }^{\left(2\right)}$ as well. By Theorem 2, our architecture accepts $\omega$ if such an edge exists. If $\omega \in \mathcal{L}\left(\widehat{G}\right)$, then the (character-level) RNGLR algorithm accepts $\omega$ as proven by Scott [17]. Our architecture accepts $\omega$, if the character-level RNGLR algorithm accepts $\omega$; therefore, if $\omega \in \mathcal{L}(\Xi )$ then our architecture accepts $\omega$. □

Traditionally, the parser is correct for a given specification $\Xi$ if the constructed SPPF is a representation of all possible derivations, such that $S\stackrel{*}{\Rightarrow }\sigma$, where $\sigma \in {\mathbf{T}}^{*}$ is a string of terminal symbols passed to the parser by the scanner. As a generalization, in our case, all possible $\sigma$ are considered. Therefore, our architecture captures all lexical ambiguities.

Theorem 4.

The parser constructs the SPPF for all possible derivations $S\stackrel{*}{\Rrightarrow }\sigma \stackrel{*}{\Rrightarrow }\omega$, where $\sigma \in {\mathbf{T}}^{*}$ and $\omega \in {\mathbf{\Omega }}^{*}$.

Proof. 

This is easy to see from Lemma 6, our architecture constructs the GSS using $\Xi$ containing all paths labeled $\iota \in {({\mathbf{N}}^{\prime }\cup \mathbf{T})}^{*}$ in the GSS constructed by the character-level RNGLR using $\widehat{G}$. The SPPF nodes are stored in the GSS edges and the construction of the SPPF nodes is equivalent as in the (character-level) RNGLR. The SPPF nodes $(A,\mathfrak{k},\mathfrak{l})$, where $A\in {\mathbf{N}}^{\prime }$, constructed by our architecture, are the same as the one constructed by the character-level RNGLR. The SPPF nodes $(t,\mathfrak{k},\mathfrak{l})$, where $t\in \mathbf{T}$, constructed by our architecture, are leaf nodes because t was recognized using the scanner. The same nodes constructed by the character-level RNGLR contain a subtree, created as a result of parsing t. Therefore, the SPPF constructed by our architecture is not exactly the same as the one for character-level RNGLR, the difference is that the subtrees for derivations $\sigma \stackrel{*}{\Rightarrow }\omega$ are missing. □

8. Experience

As our architecture utilizes context-aware scanning [6,7] and the Schrödinger’s token [3,15,16] directly, it naturally supports all of the use cases of these techniques, and the interested reader should refer to [3,6,7]. It can also resolve trivially the conflicts that are reported to be difficult to resolve using these techniques alone. These difficulties arise due to conflicts of type (II), and are exactly the ones we aimed to resolve. We will assume the scanner lookahead is not utilized, otherwise, some of these conflicts are resolved even sooner. The [6] mentions a problem of scanning keywords (or operators) that are not pairwise-prefix disjoint, for example, $\mathtt{for}$ and $\mathtt{foreach}$, appearing in C#. In the case of architecture, the automaton ${\dot{M}}_0$ recognizes both keywords. After the shorter keyword is recognized and a shift action is performed, the said automaton continues scanning and recognizes the longer keyword if possible. In that case, the first shift action is performed superfluously. The [9] mentions a problem of recognizing $\mathtt{1}..\mathtt{5}$ as two integer numbers delimited by the range operator, instead of a floating-point number followed by a dot and a cardinal number, appearing in Pascal and Ada. In the case of architecture, the automaton ${\dot{M}}_0$ recognizes both an integer and a floating-point number. After the integer number is recognized and a shift action is performed, an automaton ${\dot{M}}_1$ is added, recognizing the range operator. After the floating-point number is recognized and a shift action is performed, no automaton would recognize the dot; therefore, this shift action was performed superfluously. After the range operator is recognized and a shift action is performed ${\dot{M}}_3$ is added, again recognizing the integer or floating-point number. Note that our architecture can also handle the case where floating-point numbers are used in the range $\mathtt{1}\dots \mathtt{5}.\mathtt{0}$, which are not valid in Pascal or Ada. A similar case also occurs in COBOL, where the sentences conclude with a dot. Therefore, at the end of the sentence, $\mathtt{1}.$ should be regarded as an integer number, not a floating-point number. The [12,13] mentions a synthetic case where $\mathtt{1}.\mathtt{1}$ can be recognized either as a floating-point number or two integer numbers separated by a dot. In the case of our architecture, the automaton ${\dot{M}}_0$ first recognizes an integer number. After that, an automaton ${\dot{M}}_1$ that recognizes a dot is added, which is followed by an automaton ${\dot{M}}_2$ that recognizes an integer number. The automaton ${\dot{M}}_0$ continues scanning and recognizes the floating-point number as well. The use cases in most existing programming languages are all fairly trivial to resolve using our architecture and do not even come close to utilizing its full potential. The reason for this is that grammar writers have intentionally avoided introducing these conflicts due to the limitations of the traditional parsing architecture. Our architecture opens up new possibilities:

• Lexical syntax can vary arbitrarily for different parts of the programming language without carefully constructed scanner modes. All of the embedded sublanguages can be described in a single specification.
• During the development, thoughtless additions of regular expressions never result in an error, as our architecture accepts all possible specifications.
• A union, concatenation, and Kleene star of arbitrary specifications are invariably accepted, due to the closure properties of CFL, which allow for specifications to be composed.
• The transitions between different parts of the lexical syntax can be fluid, there is no need to surround them with sentinels. That means that multiple sublanguages can be possible in a certain context, and it can be left to the parser to determine which one it is parsing. For example, if we construct a specification that is a union of Pascal and C, the parser can determine if the input string contains Pascal or C.
• Capturing subpatterns in regular expressions are not required [37]. There is no need to treat parts of the lexical syntax as an opaque block and then extract certain parts out of it; in the case of our architecture, they can be treated as an embedded sublanguage.
• A simple error recovery resembling error tokens is supported without extensions, for example, where ; acts as a resynchronization token:

  $$\begin{array}{cc}\hfill S& ::=Ab|cb\hfill \\ \hfill A& ::=d\hfill \end{array}\begin{array}{cc}\hfill r_b& =;\hfill \\ \hfill r_c& =\neg ({\mathbf{\Omega }}^{*};)\hfill \end{array}\begin{array}{cc}\hfill r_d& =\mathtt{x}\hfill \\ \hfill r_{\$}& =\$\hfill \end{array}$$
• When writing a specification with inserted white space, multiple types of white-space can be used, for example:

  $$r_w={␣}^{*}{r}_v={␣}^{+}$$
  Even if they both occur in the same context, it does not pose a problem.

This list is of course not exhaustive. The downside of our architecture is that while it can accept any possible specification, parsing can become slow when the number of possible interpretations becomes too large. The techniques developed to resolve this issue in the scannerless architectures [4,5,27,28,29] can be applied readily and we will consider adding them in future work.

9. Related Work

The related work can be grouped into scannerless and scanner-based methods. Our architecture can be categorized into the second group. The scanner-based methods can be deterministic or non-deterministic. The former used a variety of disambiguation rules to resolve the lexical conflicts in the scanner, and the latter offload the problem to the parser. How the methods are interrelated is shown in Figure 27.

The earliest deterministic scanner-based method we are aware of is the work by Nawrocki [9]. Nawrocki categorized the types of lexical conflicts:

• The longest match conflicts, which are caused by the use of the longest match disambiguation rule.
• Identity conflicts, which are caused when the languages of the automata are not pairwise disjoint.

Additionally, lexical conflicts can occur when the languages of the automata are not prefix-free or pairwise prefix-disjoint, without causing the longest match conflicts. In that case, the longest match results in a correct parse. This was also identified by Denny [7]. Nawrocki presented two methods of resolving the lexical conflicts. The first one is called the left context $\mathrm{LC}:{\mathbf{T}}_{\$}\to \mathcal{P}\left(\mathbf{S}\right)$, which is equivalent to an inverse of the valid lookahead $\nu$. The second one is called the extended left context $\mathrm{ELC}:{\mathbf{T}}_{\$}\times {\mathbf{T}}_{\$}\to \mathcal{P}\left(\mathbf{S}\right)$, which is equivalent to an inverse of the valid lookahead–lookahead $\xi$. Because the functions are inverted, the tests $s\in \mathrm{LC}\left(t\right)$ and $s\in \mathrm{ELC}(l,l^{\prime })$ are performed instead of $t\in \nu \left(s\right)$ and $l^{\prime }\in \xi (s,l)$, which is inconsequential. The $\mathrm{LC}$ is used in the same way as in the subsequent works. When the handle-finding automaton is in state s, the symbol $t\in {\mathbf{T}}_{\$}$ is matched if $s\in \mathrm{LC}\left(t\right)$. The method is used only if a single match is found in every state; that is when $l\in {\mathbf{T}}_{\$}$, $\mathrm{LC}\left(t\right)\cap \mathrm{LC}\left(l\right)=\varnothing$. Otherwise, $\mathrm{ELC}$ is used to disambiguate the matches further. The conflicts between $t,l\in {\mathbf{T}}_{\$}$ for all $l^{\prime }\in {\mathbf{T}}_{\$}^{\prime }$ are found as in (6). The method is used only if $\mathrm{LC}\left(X\right)\cap {\bigcup }_{l^{\prime }\in {\mathbf{T}}_{\$}^{\prime }}\mathrm{ELC}(l,l^{\prime })$. When the handle-finding automaton is in state s, the symbol $l\in {\mathbf{T}}_{\$}$ is matched if $s\in {\bigcap }_{l^{\prime }\in {\mathbf{T}}_{\$}^{\prime }}\mathrm{ELC}(l,l^{\prime })$.

As identified by Denny [7] and to some extent by Keynes [8], the methods are limited. It is required that a single match is found in all handle-finding automaton states, otherwise, they cannot be used. This limitation can be lifted by further disambiguating the matched symbols using priority, as is done in context-aware scanning [6,18] and PSLR [7]. In the case of our architecture, no further disambiguation is needed.

They also cannot be used to decide when to pass a match to the parser. Let us demonstrate the problem with an example:

$$S::=abc{r}_a=/*r_b=\neg ({\mathbf{\Omega }}^{*}\ast /{\mathbf{\Omega }}^{*})r_c=\ast /r_{\$}=\$$$

(8)

The longest match conflict only really occurs when the rest of the input string is ${\omega }^{\prime }=\ast /$. If the longest match disambiguation rule is used, $M_b$ would also recognize ∗ and an error would be raised. Therefore, a match must be passed to the parser beforehand. Just the information that a conflict is possible is useless in deciding when to pass the match to the parser. This was also identified by Denny [7], although the solution of additionally providing the shortest match disambiguation rule does not solve the problem.

We used $\xi$ as an optimization, which is equivalent to $\mathrm{ELC}$ as already mentioned. We used it to construct the scanner lookahead ${\xi }^{\prime }$. Because we used one character of the scanner lookahead, our method required a buffer of one character, exactly the same as the longest match disambiguation rule. Nonetheless, our method is able to resolve the problem in the specification (8). When ∗ follows in the rest of the input, a match is passed to the parser. Note that a match is also passed to the parser for each $\ast \mathbf{\Omega }$. This is not an issue, because, without the use of this optimization, the match is always passed.

Keynes [8] presents a deterministic architecture, which uses a scanner construction approach, where the labels m and p are stored in the edges, instead of the states. The construction is equivalent to first directly constructing the automata ${({\dot{M}}_s)}_{s\in \mathbf{S}}$ for each handle-finding automaton state. Additionally, each edge $q_s\stackrel{a}{\to }{q}_s^{\prime }$ is labeled with the symbols that can be matched by starting again from the start state $l\in p(q_{s,◊}\stackrel{a}{\to }{q}_s^{″})$. Then, the lookahead symbols $t\in m(q_s\stackrel{a}{\to }{q}_s^{\prime })$ are removed if $l\notin \xi (s,t)$. That way, the scanner lookahead is incorporated into the automaton. That is possible because the labels are stored in the edges. The automata are then collapsed into a single automaton using minimization and a scanner mode is created for each of them. The scanner mode is then chosen based on the handle-finding automaton state. As noted by Keynes, the downside of the approach is that the construction method is inefficient [8]. Additionally, if we follow the disambiguation rules as given by Keynes, they cannot resolve the problem in (8). Keynes concludes that whether the matched symbol is passed to the parser or not is inconsequential as the conflict is unresolvable.

The PSLR by Denny [7] presents a deterministic architecture, which uses a scanner construction approach similar to the one used in context-aware scanning. Denny proposes a modified IELR handle-finding automaton construction method, which resolves the lexical conflicts that arise when weaker handle-finding automaton construction methods are used. The resulting handle-finding automaton has less states than the one constructed using LR(1). This would be the optimal method to use in combination with our architecture. However, it is not simple to implement; therefore, we used LALR instead.

Context-aware scanning by Van Wyk [6] and Schwerdfeger [6,18] presents the groundwork for our architecture. As already mentioned, our architecture degenerates to it when it is used for deterministic languages. We used the scanner construction approach described by Van Wyk [6] and Schwerdfeger [6,18]. Van Wyk and Schwerdfeger also suggest the architecture can be used in combination with GLR. The Tree-sitter [34] is an incremental parser generator based on this idea. However, just simply combining the approaches is not fully general.

The earliest non-deterministic method we are aware of is mentioned in the presentation of the GLR algorithm by Tomita [15,16]. As already mentioned, the lexical conflicts are treated as a parser conflict; therefore, the actions are performed for all matched symbols. The method was further discussed and termed Schrödinger’s token by Aycock and Horspool [3]. Our architecture can be viewed as a direct extension that additionally handles overlapping lexemes of different lengths. To maximize the time spent in the scanner and the length of lookahead, our architecture requires lexical feedback. As identified by Aycock and Horspool, this means that the phases cannot be parallelized. We agree that this is indeed a downside.

Keynes [8] conjectures a non-deterministic method termed generalized scanning. A generalized scanner would append the tokens to a list every time it enters a final state, and continue scanning until it enters a dead state. Then, the parser would process the tokens in the list, and the generalized scanner would move to the location of the match for the first token in the list and repeat the process. The generalized scanner would necessarily process characters multiple times in the case of lexical conflicts, and would require the entire string to be available in memory. Keynes noted that such an architecture would most likely be inefficient, and we agree with that assessment. Our architecture is a practical realization of the broader idea; for this reason, the GS in RNGSGLR stands for generalized scanning. It should be clear that our architecture does not actually work as conjectured by Keynes, as the parser “moves” with the scanner, and in that way the list is not required.

Character-level parsing was first proposed by Salomon and Cormack [4]. For this purpose, they extended the NSLR parser with exclusion and adjacency restriction disambiguation rules. NSLR is a non-canonical parser, which means that, in addition to terminal symbols, non-terminal symbols can be used as the lookahead when processing the reduce actions. Our architecture is similar in the sense that symbols in $\mathbf{T}$, which are used as lookahead, are non-terminal symbols in $\widehat{G}$. The difference is that they are recognized by the scanner instead of the parser, and that in the case of lexical conflicts, they are only recognized partially, which we termed fractional lookahead. The exclusion rules implement the differences between the languages. In the case of our architecture, the difference is only available to the regular portion of the grammar in the form of extended regular expressions. The worst-case time complexity of the parser is $\left|\omega \right|$, which is better than our architecture; however, it is not fully general, and for any given CFG, it cannot accept all corresponding character-level CFLs.

SGLR by Visser [5] is a character-level parser based on GLR. It uses the disambiguation rules introduced in character-level NSLR [4,5,38]. The parser is not non-canonical. Consequently, the reduction order is more predictable, although the worst-case complexity of the parser is the same as GLR ${\left|\omega \right|}^{\widehat{\gamma }+1}$ and our architecture. This is the only other practical approach we are aware of that is fully general, and for any given CFG can accept all corresponding character-level CFLs. To parse $\Xi$, it instead parses the corresponding character-level CFG $\widehat{G}$. Due to the use of exclusion rules in the context-free part of the grammar, it can also parse some of the context-sensitive languages. The downside of SGLR is that it uses a parser for the lexical portion of the grammar. As a result, the characters in the input string are processed multiple times—once when the shift action is performed and then again when (possibly) multiple reduce actions are performed and the parser searches for the paths passing the edge created by the shift action. In the case of our architecture, each character is processed exactly once. Since a parser is used as a scanner, the worst-case time complexity is ${\left|\omega \right|}^{\widehat{\gamma }+1}$, while, in the case of our architecture, it is ${\left|\omega \right|}^2$. The lookahead in SGLR is a single character, while, in our case, it can be multiple characters. In the case of deterministic languages, the lookahead is the whole lexeme; therefore, the same as in deterministic parsing. We only used a single character of lookahead as the scanner lookahead. The original description of the SGLR parser uses the solution by Farshi [16,39] to parse $\epsilon$-grammars. Since this solution is less efficient, the SRNGLR was developed by Economopoulos [11], which is instead based on RNGLR.

XGLR (by Begel and Graham [14]) is a non-deterministic architecture based on GLR that is the most closely related to ours. The scanner needs to be constructed manually. The grammar writer can provide multiple scanner modes. These are required to be lexically unambiguous. The scanner mode is then chosen based on the handle-finding automaton state. Whenever a lexical conflict is encountered the parser process is split, and each is associated with a scanner mode. That way, the parser processes end up in different positions when the lexemes are of different lengths [14]. Consequently, each character is processed multiple times, and the entire input string must be available in the memory. For example, if a regular expression ${\mathbf{\Omega }}^{+}$ is used in one of the modes, the rest of the input string will be traversed immediately by that parser. The others will then process the same characters again, as they follow it. In contrast, our scanner processes the characters one by one, and each is processed exactly once because the synchronization is preserved. The automaton recognizing ${\mathbf{\Omega }}^{+}$ runs alongside other automata until the end of the input string. Due to the fact that each parser process is associated with a scanner mode, this poses additional difficulties when they are later merged. The parsers also cannot share some of the data structures once they split [14]. Our architecture avoids this, as each character is processed exactly once and the synchronization is preserved. Because the parser processes are synchronized and the handle-finding automaton and the scanner states are not directly associated, no special considerations are needed when they are merged. The benefit of XGLR is that a whole lexeme is used as the lookahead, even when the grammar is heavily non-deterministic. Since XGLR uses the longest match disambiguation rule and scanner modes need to be provided manually, it is not fully general and for any given CFG cannot accept all corresponding character-level CFLs.

The LAMB and Fence by Quesada [12,13] is a non-deterministic architecture based on a chart parser that is able to resolve lexical conflicts in a similar way as a generalized scanner, as conjectured by Keynes [8]. The scanner is based on top-down regular expression recognizers, one for each terminal symbol. The recognizers are run one by one at each position $\mathfrak{i}\in \mathbf{I}$, where $\mathbf{I}\subseteq \{0,\dots ,|\omega \left|\right\}$. This means that, in the case of lexical conflicts, the characters are necessarily processed multiple times, and the entire input string must be available in the memory. For example, if a regular expression ${\mathbf{\Omega }}^{+}$ is used, the rest of the input string will be traversed from each position $\mathfrak{i}\in \mathbf{I}$. The scanner has two variants. When using the first one, at each position $\mathfrak{i}\in \mathbf{I}$ the recognizers search for the longest match, the positions of the matches are recorded, and the scanner moves to the closest one. When using the second one, at each position $\mathfrak{i}\in \mathbf{I}$, all possible matches are considered, and the recognizers are run at each position in the input string $\mathbf{I}=\{0,\dots ,|\omega \left|\right\}$. To prevent duplicates, the scanner needs to keep track of tokens that are already found at each position in the input string. The worst-case time complexity of both variants is ${\left|\omega \right|}^2$, the same as our architecture. However, the first one is not fully general and the second one always considers all interpretations as in (7), even when they are not possible; therefore, the complexity is bound to be ${\left|\omega \right|}^2$ even in the best case. The scanner generates a graph of all transitions, which is passed to the parser, where it is used to initialize the chart and the parsing commences. The input string is traversed fully by the scanner before the parsing takes place. That means the passes are completely disjoint; therefore, contextual information cannot be passed from the parser to the scanner. As a result, some of the traversals must be discarded immediately by the parser. That only occasionally happens when our architecture is used in combination with weaker handle-finding automaton construction methods, such as LALR and SLR.

The parser by Borsotti [40] is an extension of GLR that can parse the extended CFGs directly, without first translating them to CFGs. It would seem that such a parser could parse the lexical part of the grammar as well; however, this is not the case. This was also identified by the authors who stated that the parser should be used in combination with the traditional parsing architecture [40]. The parser works by constructing an automaton for each right-hand-side of the production, which is then combined into the handle-finding automaton. The handle-finding automaton also includes links in the reverse direction. When performing the reduce actions the search for the paths in the GSS is performed by following these links. Consequently, each character would be processed multiple times, once when performing the shift action, and then again when performing (possibly) multiple reduce actions.

The faster GLR parsing by Aycock and Horspool [1,41] and the generalized regular (GR) parsing by Johnstone and Scott [42] are similar to our architecture in that they improve the performance of GLR parsing using an automaton. The parsers can be used at the character level, and for any given CFG, accept all corresponding character-level CFLs (the same as our architecture). These parsers do not perform shift and reduce actions as is customary with LR parsing. Instead, the parsing process is driven mostly by the automaton and the stack is used similar to the call stack for subroutines. When searching for a non-terminal symbol, the next state is pushed on the stack, and when the non-terminal symbol is recognized, and the parser continues from the popped state. As the use of the stack is reduced, these parsers are efficient; however, as a trade-off, the automaton is prohibitively large. Our architecture completely avoids the use of the stack for the lexical syntax, otherwise it functions similar to the RNGLR parser. That way, the tables for the parser are almost the same as in RNGLR parsing and the tables for the scanner are the same as in context-aware scanning.

10. Conclusions

In this paper, we presented a novel scanner-based architecture, which, for any given CFG accepts all corresponding character-level CFLs. Our architecture lifts the restrictions of the existing scanner-based methods. It does not require any disambiguation rules for resolving lexical conflicts. However, as our architecture captures all lexical ambiguities for certain applications, disambiguation rules are still useful to limit the number of interpretations. It can directly parse all possible specifications consisting of a grammar and regular definitions. As a result, the grammar writers are not restricted to grammars where the lexical syntax can be resolved deterministically. Grammars are also easier to modify because they are not dependent on a careful selection of disambiguation rules to be parsed. Conceptually, our architecture has an unbounded parser and scanner lookahead. Our architecture is streaming, which means the characters are processed one by one and do not require a buffer.

In future work, we will attempt to incorporate the scanner lookahead into the automata, following Keynes [8], as we expect it to be more efficient. To limit the number of possible interpretations and ease the development of specifications, we will consider adding the adjacency restriction disambiguation rule [4,5,38]. We might consider other extensions, such as conjunctive and Boolean grammars, as well [43]. Our architecture can currently process symbols as they are typed at the end of the input string. Inserting them in the middle or at the beginning is currently not supported. We will attempt to create an incremental version of our architecture, which will support insertions anywhere in the input string. Our architecture could be adapted to other parsing methods that have the same power as GLR, such as GLL [44]. Since our architecture can parse any specification, it can be useful for applications where the specifications are generated procedurally, such as grammar synthesis [45]. We will also extend our metalanguage to support some of the use cases of our architecture better. That is, following [5,6,22], we will enable composing, embedding, and extending specifications.