wvabrinskas/Neuron 1.4.5

A neural network library for Swift

⭐️ 27

🕓 3 days ago

iOS macOS watchOS tvOS

.package(url: "https://github.com/wvabrinskas/Neuron.git", from: "1.4.5")

Neuron

Introduction

Neuron is a swift package I developed to help learn how to make neural networks. It is far from perfect and I am still learning. There is A LOT to learn here and I've just scratched the surface. As of right now this package provides a way to get started in machine learning and neural networks.

Support

Feel free to send me suggestions on how to improve this. I would be delighted to learn more!! You can also feel free to assign issues here as well. Run the unit tests as well to learn how the project works!

The Brain

It is fairly simple to setup the neural network Brain. This will be the only object you interface with.

Initialization

  private lazy var brain: Brain = {
    let bias = 0.01

    let nucleus = Nucleus(learningRate: 0.001,
                          bias: 0.001)
    
    let brain = Brain(nucleus: nucleus,
                      epochs: 10,
                      lossFunction: .crossEntropy,
                      lossThreshold: 0.001, 
                      initializer: .xavierNormal, 
                      gradient: .sgd)
    
    brain.add(.init(nodes: inputs, bias: bias)) //input layer

    for _ in 0..<numOfHiddenLayers {
      brain.add(.init(nodes: hidden, activation: .reLu, bias: bias)) 
    }
    
    brain.add(.init(nodes: outputs, bias: bias)) //output layer
    
    brain.add(modifier: .softmax)

    brain.add(optimizer: .adam())

    brain.logLevel = .high
    
    return brain
  }()

The Brain class is the main interactive class for dealing with the neural network. The brain object also supports different log levels so you can see what's going on in the console.

brain.logLevel = .low

  //show no logs
  case none
  
  //show only success logs
  case low
  
  //show only success and loading logs
  case medium
  
  //show all logs
  case high

Nucleus

It first takes in a Nucleus object that defines the learning rate and bias for the network.
When defining a bias it is NOT applied to the input layer.
The Nucleus object takes in 2 properties learningRate and bias
- learningRate - how quickly the node will adjust the weights of its inputs to fit the training model
  - Usually between 0 and 1.
- bias - the offset of adjustment to the weight adjustment calculation.
  - Usually between 0 and 1.

Epochs

The number of times to run through the training data.
The brain object may not hit the max number of epochs before training is finished if there is validation data passed and it reaches the defined loss threshold.

Loss Function

The loss function of the network. This will determine the loss of each epoch as well as the loss of the validation data set.

  case meanSquareError
  case crossEntropy
  case binaryCrossEntropy

Currently the network only supports Mean Squared Error, Cross Entropy and Binary Cross Entropy loss functions

Loss Threshold

The loss value the network should reach over an average of 5 epochs.

Initializer

The initializer function the brain object should use to generate the weight values for each layer.

  ///Generates weights based on a normal gaussian distribution. Mean = 0 sd = 1
  case xavierNormal

  ///Generates weights based on a uniform distribution
  case xavierUniform

Currently the network supports Xavier normal distribution and Xavier uniform distribution

Optimizer

The brain object can add an optimizer to the network by calling: brain.add(optimizer:)

public enum Optimizer {
  case adam(b1: Float = 0.9,
            b2: Float = 0.999,
            eps: Float = 1e-8)
}

Currently only the Adam optimizer is supported. There will be more soon.
You can set the various hyperparameters of the optimizer through the initilization function

Gradient Descent Optimizer

As part of the initializer of the brain object you can specify which type of gradient descent the brain performs.
By default it chooses Stochastic Gradient Descent.

public enum GradientDescent: Equatable {
  case sgd
  case mbgd(size: Int)
}

The network supports both Stochastic and Mini-Batch gradient descent.
When adding mbgd you can specify the batch size.

Adding Layers

The brain object allows for adding layers in a module way through the add function.

  public func add(_ model: LobeModel)

The LobeModel struct can be created with a simple initializer.

  public init(nodes: Int,
              activation: Activation = .none,
              bias: Float = 0, 
              normalize: Bool = false) {
    self.nodes = nodes
    self.activation = activation
    self.bias = bias
    self.normalize = normalize
  }

Nodes

the number of nodes at the layer

Activation

the activation function to be used at the layer
NOTE: If the layer is of type .input the activation function will be ignored

Bias

the bias to be added at that layer
NOTE: If the layer is of type .input the bias will be ignored

Normalize

batch normalizes the inputs to this layer

Modifiers

The network also supports adding an output activation modifier such as softmax

  public func add(modifier mod: OutputModifier) {
    self.outputModifier = mod
  }

Calling add(modifier) on the brain object will add the specified output activation to the output layer.
Currently the network on supports Softmax

  case softmax

Compiling the network

After adding all the specified layers and modifiers do not forget to call compile() on the brain object. This will connect all the layers together using the proper initializer and get the network ready for training.

Training

You can also train the Brain object by passing an expected value.

public func train(data: [TrainingData],
                    validation: [TrainingData] = [],
                    complete: ((_ complete: Bool) -> ())? = nil)

data: An array of TrainingData objects to be used as the training data set.
validation: An array of TrainingData objects to be used as the validation data set.
complete A block called when the network has finished training.

Training Data

This is the object that contains the inputs and expected output for that input

public struct TrainingData {
  public var data: [Float]
  public var correct: [Float]
  
  public init(data dat: [Float], correct cor: [Float]) {
    self.data = dat
    self.correct = cor
  }
}

Data

An array of values that should match the number of inputs into the network

Correct

A array of values that the network should target and should match the number of outputs of the network

Importing Pretrained Models

Brain can accept a pretrained model in the form of a .smodel file. Basically a renamed JSON format, this is the format the Brain object will export as well.

This will create a fully trained Brain object that is ready to go based on the trained model passed in.

 public init?(model: PretrainedModel,
              epochs: Int,
              lossFunction: LossFunction = .crossEntropy,
              lossThreshold: Float = 0.001,
              initializer: Initializers = .xavierNormal) {

PretrainedModel

The object takes in a URL to the .smodel file.

public struct PretrainedModel: ModelBuilder {
  public var fileURL: URL
  
  public init(url file: URL) {
    self.fileURL = file
  }
  ...

Exporting Pretrained Models

The Brain object can export its current weights and setup as a .smodel file. This can be used tosrc="https://raw.github.com/wvabrinskas/Neuron/master/rt later when creating a Brain object that you wish to be fully trained based on the data.

brain.exportModelURL()

Will export a URL that links to the .smodel file.

brain.exportModel()

Will export a ExportModel that describes the network

Retrieving Data

Pass in new data to feed through the network and get a result.

let out = self.brain.feed(input: data)

Returns [Float] using the new inputs and the current weights, aka. feed forward.

Data Studying

Using the Brain object you can also get the result of the loss functions of each epoch as a CSV file using the exportLoss function on Brain.

exportLoss(_ filename: String? = nil) -> URL?
filename: Name of the file to save and export. defaults to loss-{timeIntervalSince1970}
Returns the url of the exported file if successful.
Example graph:

Experimental

These features were losely tested but I have come to the conclusion that they work enough to be released. Feel free to open an issue if they do not.

GAN and WGAN support

Neuron supports Generative Adversarial Networks and Wasserstein Generative Adversarial Networks. Current Neuron doesn't support image generation with this as it doesn't support convolutional layers yet.

Implementation

Create two Brain objects. One for the discriminator and one for the generator of the network.

  private lazy var generator: Brain = {
    let bias: Float = 0
    
    let brain = Brain(learningRate: 0.00001,
                      epochs: 1)

    brain.add(.init(nodes: 7)) //some number of inputs
    brain.add(.init(nodes: 10, activation: .leakyRelu, bias: bias))
    brain.add(.init(nodes: wordLength, activation: .tanh, bias: bias))

    brain.logLevel = .none
    brain.add(optimizer: .adam())
    brain.compile() //build network

    return brain
  }()
  
  private lazy var discriminator: Brain = {
    let bias: Float = 0
    let brain = Brain(learningRate: 0.00001,
                      epochs: 1)

    brain.add(.init(nodes: wordLength))
    brain.add(.init(nodes: 10, activation: .leakyRelu, bias: bias))
    brain.add(.init(nodes: 1, activation: .sigmoid, bias: bias))
    
    brain.logLevel = .none
    brain.add(optimizer: .adam())
    brain.compile() //build network
 
    return brain
  }()

The Brain objects are neural networks of their own and will compete to minimize the selected loss function. They are created exactly as any Brain object is created as mentioned above. Only thing ignored in this initializer is the epochs. This will be handled later.
Create a GAN object using the two newly created networks.

 let gan = GAN(epochs: 1000,
                criticTrainPerEpoch: 4,
                generatorTrainPerEpoch: 1,
                gradientPenaltyCenter: 1,
                batchSize: 10)
  
  gan.add(generator: self.generator)
  gan.add(discriminator: self.discriminator)

  gan.logLevel = .none
  gan.lossFunction = .wasserstein

epochs - number of rounds of training the GAN
criticTrainPerEpoch - the number of epochs to train the critic (discriminator) per epoch of the GAN training.
generatorTrainPerEpoc - the number of epochs to train the generator per epoch of the GAN training.
gradienPenaltyCenter - (optional) when using wasserstein as the lost function the gradient penalty is centered around this value in the calculation. A good value is 1 if you're using wasserstein loss function. Default: 0.
batchSize - the number of objects to pull from the training data set per epoch.

GAN Properties

lossFunction - the loss function for the GAN to minimize. Currentlt it only supports wasserstein and minimax loss functions. Default: .minimax
randomNoise: () -> [Float] - a block that the GAN object will call every epoch to get input data for the generator. The generator takes in random noise from a described latent space. This block is used to define that latent space.
validateGenerator: (_ output: [Float]) -> Bool - a block that the GAN checks every 5 epochs to determine if it has reached it's desired output and to stop training. Returning true here will stop training. This block is mostly irrelevant as GAN training usually cannot be stopped programmatically. The output is the output vector from the generator at the current epoch.
discriminatorNoiseFactor - this value is mostly useful for minimax loss function. This introduces a noise to the correct label. A value greater than 0 but less than 1 is acceptable. This will assign the correct label a value a random number based on the factor.
```
let factor = min(1.0, max(0.0, noise))
let min = min(label, abs(label - factor))
let max = max(label, abs(label - factor))

let newLabel src="https://raw.github.com/wvabrinskas/Neuron/master/at.random(in: (min...max))
```

Usrc="https://raw.github.com/wvabrinskas/Neuron/master/

To train the GAN you just call train on the GAN object you created with an array of TrainingData objects. Similar to any Brain object.

self.ganBrain.train(data: trainingData) { success in }

To get generated data from the generator of the GAN you just call getGeneratedSample on your GAN object. This returns an array of Float from the generator.

let result = self.ganBrain.getGeneratedSample()

You can also get data from the disrimanator to discriminate against some input data.

let result = self.ganBrain.discriminate(input)

GAN Data Analysis

A GAN will attempt to map between one distrubition to another. You can see below the input distribution is a normal even distribution while the output distribution from the GAN is a gaussian distribution.

TODOs

GPU Acceleration is still in the works.
Convolutional layer support
Much more...

Resources

Cross Entropy + Softmax

Videos:

https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/softmax

Cost Function vs Loss Function:

Gradient Clipping:

https://towardsdatascience.com/what-is-gradient-clipping-b8e815cdfb48

Activation Functions:

https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6

Backpropagation:

Validation:

Classification:

https://becominghuman.ai/making-a-simple-neural-network-classification-2449da88c77e

Weight Initialization

https://prateekvishnu.medium.com/xavier-and-he-normal-he-et-al-initialization-8e3d7a087528

Optimizers

Swiftpack.co - wvabrinskas/Neuron as Swift Package