Post cover image

June 20, 2025

Designing Effective Tree-sitter Grammars

Writing a grammar for Tree-sitter involves more than simply translating an existing context-free grammar (CFG). It requires a thoughtful…

By Lince Mathew

3 min read

Writing a grammar for Tree-sitter involves more than simply translating an existing context-free grammar (CFG). It requires a thoughtful design that balances expressiveness, readability, and parsing efficiency. This article walks through the key ideas and practical examples for crafting a Tree-sitter grammar.

Starting with a Good Structure

A well-structured grammar helps Tree-sitter produce useful syntax trees. Ideally, each node in the tree should correspond closely to a recognizable construct in the source language. This ensures the tree is easy to analyze and manipulate.

In addition, Tree-sitter performs best with grammars that are close to the LR(1) class. Though Tree-sitter uses the GLR algorithm and supports general CFGs, optimizing for LR(1)-like rules reduces ambiguity and improves performance.

Initializing Grammar Rules

A common starting point is to group language constructs into familiar categories: declarations, definitions, statements, expressions, types, and patterns. Here's an example structure for a simple Go-like language:

rules: {
  source_file: $ => repeat($._definition),
  _definition: $ => choice(
      $.function_definition
    ),
    function_definition: $ => seq(
      'func',
      $.identifier,
      $.parameter_list,
      $._type,
      $.block
    ),
    parameter_list: $ => seq('(', ')'),
    _type: $ => choice('bool'),
    block: $ => seq('{', repeat($._statement), '}'),
    _statement: $ => choice($.return_statement),
    return_statement: $ => seq('return', $._expression, ';'),
    _expression: $ => choice($.identifier, $.number),
    identifier: $ => /[a-z]+/,
    number: $ => /\d+/
  }

The first rule in the rules object becomes the grammar's entry point. This skeleton gives a clear overview and makes it easier to extend specific parts incrementally.

Building Subsystems

Once the basic layout is in place, you can add more detail. Here's an example of expanding the type system:

_type: $ => choice(
  $.primitive_type,
  $.array_type,
  $.pointer_type
),
primitive_type: $ => choice('bool', 'int'),
array_type: $ => seq('[', ']', $._type),
pointer_type: $ => seq('*', $._type)

You can build similar subsystems for expressions, statements, or any other construct.

Flattening Deep Expression Trees

Language specs often describe expression precedence with deeply nested rules. For example, itx + y may be represented through many layers of expression types. Instead of replicating this depth, you can define a flatter structure:

_expression: $ => choice(
  $.identifier,
  $.unary_expression,
  $.binary_expression
),
unary_expression: $ => choice(
  seq('-', $._expression),
  seq('!', $._expression)
),
binary_expression: $ => choice(
  seq($._expression, '+', $._expression),
  seq($._expression, '*', $._expression)
)

However, this flattening introduces ambiguity. Tree-sitter needs help resolving which operations bind more tightly.

Managing Precedence

Use the prec function to indicate how tightly rules should bind:

unary_expression: $ => prec(2, choice(
  seq('-', $._expression),
  seq('!', $._expression)
)),
binary_expression: $ => choice(
  prec.left(2, seq($._expression, '*', $._expression)),
  prec.left(1, seq($._expression, '+', $._expression))
)

prec.left ensures left-associativity. This lets Tree-sitter interpret a * b * c as (a * b) * c.

Handling Ambiguity with Conflicts

Some language features are inherently ambiguous. For example, in JavaScript, [x, y] could be an array or a destructuring pattern. Tree-sitter supports multiple interpretations:

conflicts: $ => [
  [$.array, $.array_pattern]
]

Declaring such conflicts explicitly allows Tree-sitter to explore both interpretations during parsing.

Hidden Rules

Rules prefixed with an underscore (e.g. _expression) are hidden in the resulting syntax tree. These are useful for intermediate or grouping rules that do not represent meaningful constructs by themselves.

Naming Child Nodes with Fields

Fields help identify specific parts of a syntax node by name:

function_definition: $ => seq(
  'func',
  field('name', $.identifier),
  field('parameters', $.parameter_list),
  field('return_type', $._type),
  field('body', $.block)
)

This allows later code to retrieve children by name instead of relying on their position.

Lexing in Tree-sitter

Parsing involves tokenizing source code into meaningful pieces. Tree-sitter does this during parsing (on-demand). It uses several rules to resolve which token to choose when multiple tokens could match:

  • Context-awareness: Only tokens valid in the current context are considered.
  • Precedence: Tokens with higher precedence are preferred.
  • Length: Longer matches win over shorter ones.
  • Specificity: Strings take priority over regexes.
  • Rule order: Earlier rules win if all else is equal.

Keywords and Word Tokens

Languages often have both keywords (like return) and general identifiers (like variable names). To resolve ambiguities, Tree-sitter allows defining a "word" token:

word: $ => $.identifier

This makes Tree-sitter recognize keyword boundaries and improves both accuracy and performance. For instance, it ensures instanceofSomething isn't mistaken for the keyword instanceof.

Conclusion

These grammars not only improve editor features like syntax highlighting and folding but also enable powerful static analysis and tooling capabilities.

PRO TIP: Struggling to locate API endpoints across your codebase?

LiveAPI lets you instantly find and explore every endpoint across all your repositories — give it a try and see how much easier navigation can be!

None