Swiftpack.co - aespinilla/GPT3-Tokenizer as Swift Package

Swiftpack.co is a collection of thousands of indexed Swift packages. Search packages.
See all packages published by aespinilla.
aespinilla/GPT3-Tokenizer 0.1.2
GPT3 encoder & decoder tool written in Swift
⭐️ 18
🕓 2 weeks ago
iOS macOS watchOS tvOS
.package(url: "https://github.com/aespinilla/GPT3-Tokenizer.git", from: "0.1.2")

GPT3 Tokenizer

GPT3 encoder & decoder tool written in Swift


GPT-2 and GPT-3 use byte pair encoding to turn text into a series of integers to feed into the model. This is a Swift implementation of OpenAI's original python encoder/decoder which can be found here and based on this Javascript implementation here

Install with Swift Package Manager

Add this repository url to Swift Package Manager



import GPT3_Tokenizer

let gpt3Tokenizer = GPT3Tokenizer()

let text = "hello 👋 world 🌍"

// Encode
let encoded = gpt3Tokenizer.encoder.enconde(text: text)

// Decode
let decoded = gpt3Tokenizer.decoder.decode(encode: encoded)


Stars: 18
Last commit: 4 days ago
jonrohan Something's broken? Yell at me @ptrpavlik. Praise and feedback (and money) is also welcome.

Release Notes

2 weeks ago
  • Improve performance on encoder and decoder

Swiftpack is being maintained by Petr Pavlik | @ptrpavlik | @swiftpackco | API | Analytics