Swiftpack.co -  wojtekcz/language2motion as Swift Package
Swiftpack.co is a collection of thousands of indexed Swift packages. Search packages.
wojtekcz/language2motion
The goal of this project is to create multi-modal implementation of Transformer architecture in Swift.
.package(url: "https://github.com/wojtekcz/language2motion.git", from: "v0.3.0")

Language2motion

The goal of this project is to create multi-modal implementation of Transformer architecture in Swift. It's a learning exercise for me, so I've taken it slowly, starting from simple image classifier and building it up.

Also it's an attempt to answer the question if Swift for Tensorflow is ready for non-trivial work.

The use-case is based on a paper "Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks" by Matthias Plappert. He created a nice dataset of few thousand motions "The KIT Motion-Language Dataset (paper)", website.

The Motion2Language Transformer which works is there, already. Lang2motion one started to work recently, and I'm implementing more sophisticated motion generation strategy now.

I'm using modified Swift Transformer implementation by Andre Carrera.

Models

  • motion 2 language
    • ☑ Transformer from motion to annotation
  • language 2 motion
    • ☑ Transformer from annotation to motion

Dataset files

Motion player

Runtime env

GitHub

link
Stars: 7
Last commit: 2 weeks ago

Ad: Job Offers

iOS Software Engineer @ Perry Street Software
Perry Street Software is Jack’d and SCRUFF. We are two of the world’s largest gay, bi, trans and queer social dating apps on iOS and Android. Our brands reach more than 20 million members worldwide so members can connect, meet and express themselves on a platform that prioritizes privacy and security. We invest heavily into SwiftUI and using Swift Packages to modularize the codebase.

Submit a free job ad (while I'm testing this). The analytics numbers for this website are here.

Dependencies

Release Notes

third set of datasets
1 year ago

Scaled motions, using simplified MotionSample, and actual motion recording frequency when downsampling.

Swiftpack is being maintained by Petr Pavlik | @ptrpavlik | @swiftpackco | API | Analytics