I am proposing a system in which one separates the question of "is this definition in scope" from "elaborate this definition".

In my language, a project is a set of files, with a distinguished entry point. Files may import other files, but there must be no cycles. Imports are transitive: if A imports B, and B imports C, then A's definitions have access to those in C. The transitive closure of a directed acyclic graph is a partial order.

Within a file, we have a list of commands, such as def, axiom, inductive. A command may produce one or more constants. For example, inductive Nat | ... produces four constants: the type former, zero, successor, and the dependent eliminator.

What is not obvious is holes. In my elaborator, a hole, produced by the keyword sorry, is elaborated by creating a new axiom on the fly and abstracting over the context. So, if a sorry is written in a place which expects a Nat, given that x : Nat is in scope, it will generate something like

and then elaborate the sorry to foo._sorry.1 x.

In practice, the name of the axiom is supposed to be inaccessible. Moreover, it should be unique in the whole project, and it must also be something that is a memoisable result. In Lean, uniqueness is achieved by tagging the sorry's source location to it. In my elaborator, I use a more abstract notion of "path" in the AST, which achieves a similar thing, but is stable under whitespace changes and reordering of commands within a file, hence memoisable.

Suppose we are elaborating foo, and want to fetch a constant bar. We first have to determine: hypothetically, in a batch elaborator, would bar be in scope at the point of foo? The answer is given as follows. Determine where foo and bar are in the whole project (file and command index). If foo and bar are in the same file, does bar come before foo? If not, does foo's file transitively import bar's file? This requires an adjacency matrix of the transitive closure of the import graph. Luckily, we only need to do this query once per run, as it is always memoised for the second time onwards. So in some sense, the query-based elaborator is merely pretending to be like a batch elaborator.

Fun little Lean proof that Player 1 is winning in Chomp. More generally:

A subset S of a poset is downward closed if for all x \in S and y \le x, we have y \in S. A board is a finite, downward-closed subset of a poset that contains a bottom element. A move consists of choosing a non-bottom element p of the board, and removing all elements q \ge p. The game ends when only the bottom element remains, and the player who made the last move wins.

If the board has a top element, then Player 1 wins. The proof is by a well-known strategy-stealing argument.

The AST and CST are homogenous and the AST is indexed by its path in the tree. By homogenous, I mean that there is only one kind of internal node, and it carries a SyntaxNodeKind, and an Array of children. Lean.SyntaxNodeKind is really just the same as Lean.Name. By using Lean.SyntaxNodeKind, I can piggy back off of Lean's parser even more: translating Lean's concrete syntax tree to mine is a five-line recursive function.

Paths exist to have a stable notion of "relative location" without using absolute character positions in source files. Without a homogeneous tree, whitespace-only changes can cause cache invalidation. Paths are defined as List Nat, so a path of [2, 0, 1] means: "from the syntax node of this definition, take the third child, first child, second child." For example: (def foo^0 : Nat^1 := ((Nat.add^0 Bool.true^1)^0 Nat.zero^1)^2) identifies Bool.true relative to the beginning of def foo. It does not matter if foo is reordered in the file, or if other definitions are added before it. The type checker functions all have access to the current path in the AST, so error diagnostics can be emitted with a relative location.

Now it carries its path as an index:

The reason for making the path intrinsic in the AST type is that when recursively descending into a subterm e[i], someone has to update the context and cons i onto the path. This is prone to bugs. If the path is part of the type, the unifier does this automatically.

The CST is a green tree in the sense of Roslyn and rust-analyzer: it stores no absolute positions, only token text. Positions are computed by summing widths when traversing the tree. This makes the CST stable under edits in other parts of the file.

I have plans to make the parser output provably correct green concrete syntax trees.

To elaborate a definition, you generally need to fetch its dependencies, which are often just previous definitions. Unfortunately this makes memoisation hard.

Suppose you are elaborating foo. The naive way would be to include "the entire global environment, prior to foo" in the dependencies of foo. This is very easy to invalidate. If one writes another definition before foo, then foo's dependencies get invalidated.

A less naive way is to track only those definitions which foo actually depends on. If foo's definition depended on bar and baz, then as long as those have not changed, foo ought to stay the same. This falls into several traps.

Suppose you have

and then you swap the lines in the editor:

A batch elaborator would process foo and reject it, as bar is not defined. A query-based elaborator might check whether foo was memoised, which it indeed is, and then check bar is defined, which it indeed is, and accept.

Here is another example:

changed to

A batch elaborator would reject bar, as foo is not defined. A query-based elaborator would check bar, see that its definition changed, and then check foo. Then it would have to check whether foo's dependencies changed, which includes bar. This is a cycle. Luckily, cycles can be caught in a query system. However, this is not exactly elegant.

Bidirectional type checking now works for this Tarski universe. All that has to be done is check whether a piece of syntax appears in term position or type position, and elaborate accordingly.

The do_el function implements the decoding of type codes during evaluation. When evaluating TyEl t, if t evaluates to a type code like VTmPiHat, it unfolds to the corresponding semantic type VTyPi. This implements the Tarski equations like El(π(a, b)) ≡ Π(El(a), El(b)).

For NbE, the semantic domains vl_ty and vl_tm represent weak head normal forms. A neutral is a head (a local or global variable) applied to a spine of eliminators (applications, projections), i.e. essentially a stuck computation that cannot reduce any further. The neutral type is shared between terms and types.

The only irreducible El is one applied to a neutral, VTyEl of neutral. Closures defer evaluation under binders until instantiation.

Integrated Jane Street's Incremental library as a proof of concept for reactive elaboration. The pipeline now watches source files and recomputes only invalidated stages when the source changes.

The architecture wraps each compilation stage (lexing, parsing, elaboration) as an incremental node with cutoffs based on structural equality. Changes propagate through the dependency graph, and observers log when stages are initialised, changed, or invalidated.

Currently the incrementality is at the file level only, because I am yet to figure out how to construct dependency graphs at runtime, probably using the bind functions.

Note: ollef/sixty builds its query engine on the Haskell library Rock, which defines queries with GADTs.

Fixed a bug in the pattern unification algorithm. The spine inversion was processing arguments in the wrong order, causing incorrect variable mappings.

Added tests for the elaborator using Alcotest. Terms are compared up to structural equality, ignoring the variable names kept around for pretty printing.

I have added bisect_ppx to measure test coverage.

The elaborator now handles metavariable inference in application contexts. When type checking id _ (), the hole _ in the type argument position gets correctly unified with Unit based on the argument type. This works through the application inference rules in elab.ml:

    let f', f_ty = infer ctx names f in
    let a_ty, b_clos =
    match Eval.force f_ty with
    | VPi (_, a, b) -> (a, b)
    | _ ->
    (* Not a Pi - insert metas and unify *)
    let a_val = Eval.eval ctx.env (fresh_meta_ctx ctx) in
    let b_tm = fresh_meta_ctx (Check.bind_var ctx a_val) in
    let pi_ty = VPi ("_", a_val, Closure (ctx.env, b_tm)) in
    unify_catch ctx pi_ty f_ty;
    (* ... *)]]>

When the function type is not immediately a Pi type, the elaborator inserts metavariables for both the domain and codomain, then unifies. check ctx names a a_ty checks the argument against the expected type, and unification propagates constraints back to the metavariables.

Values need to be forced before pattern matching. The type system represents metavariables lazily (VFlex), so after inference you get something that evaluates to VUnit but isn't immediately VUnit. The Eval.force function follows the metavariable solution chain:

 (
    match meta m with
    | VFlex (m', sp') -> force (apply_spine (VFlex (m', sp')) sp)
    | v -> force (apply_spine v sp))
    | v -> v]]>

Metavariables need to be fully instantiated before structural inspection.

The elaborator seems to be in a reasonable state for simple dependent types.

1. Identifying the natural query boundaries (probably at definition boundaries, and maybe at subterm granularity)
2. Making the context explicit and immutable for each query
3. Representing source positions and dependencies explicitly

I started my Part II Project, titled Query-based incremental Elaborator for a Dependently Typed Language. The project is supervised by Jon Sterling. To summarise, the goal of the project is implement an incremental, query-based elaborator for a dependently typed toy language, for efficient type checking in interactive development environments, such as proof assistants. The motivation for this, is that traditional type checkers for dependent types operate in batch mode, re-checking entire programs after each change. The project will investigate bringing query-based compilation techniques to dependent type elaboration.

A query-based architecture decomposes type checking into queries, memoisable functions whose only effect is to fetch the results of other queries. By tracking dependencies for these queries, we can save a lot of work by recomputing only the affected parts of the type checking process.