Swiftpack.co - ChimeHQ/Neon as Swift Package

Swiftpack.co is a collection of thousands of indexed Swift packages. Search packages.
See all packages published by ChimeHQ.
ChimeHQ/Neon 0.4.0
A system for highlighting, indenting, and querying syntax on macOS and iOS
⭐️ 145
🕓 6 days ago
iOS macOS
.package(url: "https://github.com/ChimeHQ/Neon.git", from: "0.4.0")

Build Status License Platforms

Neon

A Swift library for efficient highlighting, indenting, and querying the structure of language syntax.

It features:

  • Minimal text invalidation
  • Support for multiple token sources
  • A hybrid sync/async system for targeting flicker-free styling on keystrokes
  • tree-sitter integration
  • Compatibility with lazy text data reading
  • Flexibility when integrating with a larger text system

It does not feature:

  • A theme system
  • A single View subclass
  • Low complexity

Neon has a strong focus on efficiency and flexibility. These qualities bring some serious complexity. Right now, Neon is a collection of components that can be assembled together as part of a larger text system. It does not include a single component that ties everything together.

I realize that many people are looking for exactly that. But, it's deceptively difficult, as text systems can be phenomenally complicated. I'd love to make easier-to-use parts, and that's a goal. But, it has to be done in a way that does not sacrifice flexibility.

The library is being extracted from the Chime editor. It's a big system, and pulling it out is something we intend to do over time.

Why is this so complicated?

Working with small, static, and syntactically correct documents is one thing. Achieving both high performance and high quality behavior for an editor is totally different. Work needs to be done on every keystroke, and minimizing that work requires an enormous amount of infrastructure and careful design. Before starting, it's worth seriously evaluating your performance and quality needs. You may be able to get away with a much simpler system. A lot of this boils down to size of the document. Remember: most files are small, and small files can make even the most naive implementation feel acceptable.

Some things to consider:

  • Latency to open a file
  • Latency to visible elements highlight
  • Latency to end-of-document highlight
  • Latency on keystroke
  • Precise invalidation on keystroke
  • Highlight quality in the face of invalid syntax
  • Ability to apply progressively higher-quality highlighting
  • Precise indentation calculation

Not all of these might matter you. Neon's components are fairly loosely-coupled, so maybe just one or two parts might be usable without the whole thing.

Language Support

Neon is built around the idea that there can be multiple sources of information about the semantic meaning of the text, all with varying latencies and quality.

  • Language Server Protocol has semantic tokens, which is high quality, but also high latency.
  • tree-sitter is very good quality, and can potentially be low-latency
  • Regex-based systems can have ok quality and low-latency
  • Simpler pattern-matching systems generally have poor quality, but have very low latency

Neon includes built-in support for tree-sitter via SwiftTreeSitter. Tree-sitter also uses separate compiled parsers for each language. Thanks to tree-sitter-xcframework, you can get access to pre-built binaries for the runtime and some parsers. It also includes the needed query definitions for those languages. This system is compatible with parsers that aren't bundled, but it's definitely more work to use them.

Integration

Neon's components need to react to various events:

  • the text is about to change
  • the text has changed
  • a text change has been processed and is now ready to be styled
  • the visible text has changed
  • the styling has become invalid (ex: the theme has changed)

How and where they come from depends on your text setup. And, not every component needs to know about all of these, so you may be able to get away with less.

Simple Integration

A minimal integration can be achieved by configuring a Highlighter to interface with an NSTextView's text container:

func applicationDidFinishLaunching(_ aNotification: Notification) {
   guard let textContainer = textView.textContainer, let textStorage = textView.textStorage else {
      preconditionFailure()
   }
   let textInterface = TextContainerSystemInterface(textContainer: textContainer, attributeProvider: self.attributeProvider)
   self.highlighter = Highlighter(textInterface: textInterface, tokenProvider: self.tokenProvider)
   textStorage.delegate = self
   self.highlighter.invalidate()
}

Attaching the highlighter to a text view interface tells it what to update, but not when. You have to notify the highlighter whenever the text view's content changes, and invalidate existing highlighting as needed. Such notifications can be conveyed by making yourself the delegate of your text view's textStorage, and implementing this delegate method:

func textStorage(_ textStorage: NSTextStorage, didProcessEditing editedMask: NSTextStorageEditActions, range editedRange: NSRange, changeInLength delta: Int) {
   // Map NSTextStorageDelegate editedRange to Neon's style of editedRange
   let adjustedRange = NSRange(location: editedRange.location, length: editedRange.length - delta)
   self.highlighter.didChangeContent(in: adjustedRange, delta: delta)

   DispatchQueue.main.async {
      self.highlighter.invalidate()
   }
}

Notice that the invalidate method is dispatched asynchronously to ensure the styles are not updated until after the underlying text storage is done being edited.

The initial configuration of highlighter included references to self.tokenProvider and self.attributeProvider, which are responsible for providing the logic behind what gets highlighted, and how it should be done. At a minimum, the TokenProvider generates and supplies named tokens that correspond to ranges of text:

let paintItBlackTokenName = "paintItBlack"

func tokenProvider(_ range: NSRange, completionHandler: @escaping (Result<TokenApplication, Error>) -> Void) {
   var tokens: [Token] = []
   guard let searchString = self.textView.textStorage?.string else {
      // Could also complete with .failure(...) here
      completionHandler(.success(TokenApplication(tokens: tokens, action: .replace)))
      return
   }

   if let regex = try? NSRegularExpression(pattern: "[^\\s]+\\s{0,1}") {
      regex.enumerateMatches(in: searchString, range: range) { regexResult, _, _ in
         guard let result = regexResult else { return }
         for rangeIndex in 0..<result.numberOfRanges {
            let tokenRange = result.range(at: rangeIndex)
            tokens.append(Token(name: paintItBlackTokenName, range: tokenRange))
         }
      }
   }

   completionHandler(.success(TokenApplication(tokens: tokens, action: .replace)))
}

In this trivial example, the "paint it black" token is unilaterally applied to every non-whitespace range of the text. It demonstrates how you use a token provider to associate named tokens with arbitrary ranges of text. It's important to understand though that supplying the token doesn't change anything about the appearance of the corresponding text. In order to achieve that, you need to implement the attribute provider, which effectively translates named tokens in to suitable attributes:

func attributeProvider(_ token: Token) -> [NSAttributedString.Key: Any]? {
   if token.name == paintItBlackTokenName {
      return [.foregroundColor: NSColor.red, .backgroundColor: NSColor.black]
   }
   return nil
}

Now our example achieves its goal of "painting black" any runs of non-whitespace characters, along with single whitespace characters between them:

Screenshot of 'Paint it Black' text window showing text with a black background and red text.

Using this basic structure you can annotate the text with tokens while separately determining the appropriate styling for those tokens.

Advanced Integration

Achieving better performance and guaranteed flicker-free highlighting is more challenging. Monitoring the visible rect of the NSTextView will improve performance. You need to know when a text change has been processing by enough of the system that styling is possible. This point in the text change lifecycle is not natively supported by NSTextStorage or NSLayoutManager. It requires an NSTextStorage subclass. But, even that isn't quite enough unfortunately, as you still need to precisely control the timing of invalidation and styling. This is where HighlighterInvalidationBuffer comes in. I warned you this was complicated.

Relationship to TextStory

TextStory is a library that contains three very useful components when working with Neon.

  • TSYTextStorage gets you all the text change life cycle hooks without falling into the NSString/String bridging performance traps
  • TextMutationEventRouter makes it easier to route events to the components
  • LazyTextStoringMonitor allows for lazy content reading, which is essential to quickly open large documents

You can definitely use Neon without TextStory. But, I think it may be reasonable to just make Neon depend on TextStory to help simplify usage.

Components

Highlighter

This is the main component that coordinates the styling and invalidation of text.

  • Connects to a text view via TextSystemInterface
  • Monitors text changes and view visible state
  • Gets token-level information from a TokenProvider

Note that Highlighter is built to handle a TokenProvider calling its completion block more than one time, potentially replacing or merging with existing styling.

HighlighterInvalidationBuffer

In a traditional NSTextStorage/NSLayoutManager system (TextKit 1), it can be challenging to achieve flicker-free on-keypress highlighting. This class offers a mechanism for buffering invalidations, so you can precisely control how and when actual text style updates occur.

TextContainerSystemInterface (macOS only)

An implementation of the TextSystemInterface protocol for an NSTextContainer-backed NSTextView. This takes care of the interface to NSTextView and NSLayoutManager, but defers Token-style translation (themes) to an external AttributeProvider.

TreeSitterClient

This class is an asynchronous interface to tree-sitter. It provides an UTF-16 code-point (NSString-compatible) API for edits, invalidations, and queries. It can process edits of String objects, or raw bytes. Invalidations are translated to the current content state, even if a queue of edits are still being processed. It is fully-compatible with reading the document content lazily.

  • Monitors text changes
  • Can be used to build a TokenProvider
  • Requires a function that can translate UTF-16 code points to a tree-sitter Point (line + offset)

TreeSitterClient provides APIs that can be both synchronous, asynchronous, or both depending on the state of the system. This kind of interface can be important when optimizing for flicker-free, low-latency highlighting live typing interactions like indenting.

Using it is quite involved - here's a little example:

import SwiftTreeSitter
import tree_sitter_language_resources
import Neon

// step 1: setup

// construct the tree-sitter grammar for the language you are interested
// in working with manually
let unbundledLanguage = Language(language: my_tree_sitter_grammar())

// .. or grab one from tree-sitter-xcframework
let swift = LanguageResource.swift
let language = Language(language: swift.parser)

// construct your highlighting query
// this is a one-time cost, but can be expensive
let url = swift.highlightQueryURL!
let query = try! language.query(contentsOf: url)

// step 2: configure the client

// produce a transformer function that can map UTF16 code point indexes to Point (Line, Offset) structs
let transformer: Point.LocationTransformer = { codePointIndex in 
   return nil // Should return "Point" in text layout
}
      
let client = TreeSitterClient(language: language, transformer: transformer)

// this function will be called with a minimal set of text ranges
// that have become invalidated due to edits. These ranges
// always correspond to the *current* state of the text content,
// even if TreeSitterClient is currently processing edits in the
// background.
client.invalidationHandler = { textTarget in
   // TextTarget is a set, range, or .all
}

// step 3: inform it about content changes
// these APIs match up fairly closely with NSTextStorageDelegate,
// and are compatible with lazy evaluation of the text content

// call this *before* the content has been changed
client.willChangeContent(in: range)

// and call this *after*
client.didChangeContent(to: string, in: range, delta: delta, limit: limit)

// step 4: run queries
// you can execute these queries directly in the invalidationHandler, if desired

// Many tree-sitter highlight queries contain predicates. These are both expensive
// and complex to resolve. This is an optional feature - you can just skip it. Doing
// so makes the process both faster and simpler, but could result in lower-quality
// and even incorrect highlighting.

let provider: TreeSitterClient.TextProvider = { (range, _) -> String? in
   return nil
}

let range = NSMakeRange(0, 10) // for example
client.executeHighlightsQuery(query, in: range, textProvider: provider) { result in
   // Token values will tell you the highlights.scm name and range in your text
}

Suggestions or Feedback

We'd love to hear from you! Get in touch via twitter, an issue, or a pull request.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

GitHub

link
Stars: 145
Last commit: 6 days ago
jonrohan Something's broken? Yell at me @ptrpavlik. Praise and feedback (and money) is also welcome.

Release Notes

v0.4.0
6 days ago
  • More detailed docs for TokenProvider
  • take the token ranges themselves into account when applying styles
  • edited ranges should include adjacent characters

Swiftpack is being maintained by Petr Pavlik | @ptrpavlik | @swiftpackco | API | Analytics