As a research interest, I play with evolutionary algorithms a lot. Recently I’ve been messing around with Neural Nets that are evolved rather than trained with backpropagation.
Because this is a blog post, and to further demonstrate that literally anything can result in evolution, I’m going to be using a hill climbing algorithm. Here’s the gist of it.
- Initially, we will start with a Neural Network with random weights.
- We’re going to clone the network, pick a weight and change it to a random number.
- Evaluate the old network and the new network and get their scores
- If the new network has done better or the same as the old one, replace the old network with it
- Repeat until the results are satisfactory
The algorithm
The algorithm is shown below. All it does is split the given data into training and test parts, randomly change the neural network weights until the score improves, and then use the test data to determine how good we did.
def train_and_test(X, y, nn_size, iterations=1000, test_size=None, stratify=None):
random.seed(445)
np.random.seed(445)
net = NeuralNetwork(nn_size)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=test_size, random_state=42, stratify=stratify
)
score = 0
for i in range(iterations):
score = net.get_score(X_train, y_train)
new = net.clone()
new.mutate()
new_score = new.get_score(X_train, y_train)
if new_score >= score:
net = new
score = new_score
print(f"Training set: {len(X_train)} elements. Error: {score}")
score = net.get_classify_score(X_test, y_test)
print(f"Test set: {score} / {len(X_test)}. Score: {score / len(X_test) * 100}%")
Iris flower dataset
If you are learning about classifiers, the Iris flower dataset is probably the first thing you’re going to test. It is like the “Hello World” of classification basically.
The dataset includes petal and sepal size measurements from 3 different Iris species. The goal is to get measurements and classify which species they are from.
You can find more information on the dataset here.
data = pandas.read_csv("IRIS.csv").values
name_to_output = {
"Iris-setosa": [1, 0, 0],
"Iris-versicolor": [0, 1, 0],
"Iris-virginica": [0, 0, 1],
}
rows = data.shape[0]
data_input = data[:, 0:4].reshape((rows, 4, 1)).astype(float)
data_output = np.array(list(map(lambda x: name_to_output[x], data[:, 4]))).reshape(
(rows, 3)
)
train_and_test(data_input, data_output, (4, 4, 3), 10000, 0.2)
Training set: 120 elements. Error: -5.697678436657024
Test set: 29 / 30. Score: 96.66666666666667%
96% accuracy isn’t bad such a simple algorithm. But it has that accuracy when it trains with 120 samples and tests with 30. Let’s see if it’s good at generalization by turning our train/test split into 0.03/0.97.
As you can see below; just by training on 4 samples, our network is able to classify the rest of the data with a 94% accuracy.
train_and_test(data_input, data_output, (4, 4, 3), 10000, 0.97)
Training set: 4 elements. Error: -0.8103166051741318
Test set: 138 / 146. Score: 94.52054794520548%
Cancer diagnosis dataset
This dataset has includes some data/measurements about tumors, and classifies them as either Benign (B) or Malignant (M).
You can find the dataset and more information about it here.
data = pandas.read_csv("breast_cancer.csv").values[1:]
rows = data.shape[0]
name_to_output = {"B": [1, 0], "M": [0, 1]}
data_input = data[:, 2:32].reshape((rows, 30, 1)).astype(float) / 100
data_output = np.array(list(map(lambda x: name_to_output[x], data[:, 1]))).reshape(
(rows, 2)
)
train_and_test(data_input, data_output, (30, 30, 15, 2), 10000, 0.3)
Training set: 397 elements. Error: -5.626705318006574
Test set: 159 / 171. Score: 92.98245614035088%
To see if the network is able to generalize, let’s train it on 11 samples and test it on 557. You can see below that it has an 86% accuracy after seeing a tiny amount of samples.
train_and_test(data_input, data_output, (30, 30, 15, 2), 10000, 0.98)
Training set: 11 elements. Error: -0.2742514647152907
Test set: 481 / 557. Score: 86.35547576301616%
Glass classification dataset
This dataset has some material measurements, like how much of each element was found in a piece of glass. Using these measurements, the goal is to classify which of the 8 glass types it was from.
This dataset doesn’t separate cleanly, and there aren’t a lot of samples you get. So I cranked up the iteration number and added more hidden layers. Deep learning baby!
You can find more information on the dataset here.
data = pandas.read_csv("glass.csv").values[1:]
rows = data.shape[0]
data_input = data[:, :-1].reshape((rows, 9, 1)).astype(float)
data_output = np.array(list(map(lambda x: np.eye(8)[int(x)], data[:, -1]))).reshape((rows, 8))
train_and_test(data_input, data_output, (9, 9, 9, 9, 8), 20000, 0.3, stratify=data_output)
Training set: 149 elements. Error: -8.261249669954738
Test set: 47 / 64. Score: 73.4375%
After I saw this result, I wasn’t super thrilled about it. But after I went through the other solutions on Kaggle and looked at their results, I found out that this wasn’t bad compared to other classifiers.
But where’s the Neural Network code?
Here it is. While it’s a large chunk of code, I find that this is the least interesting part of the project. This is basically a bunch of matrices getting multiplied and mutated randomly. You can find a bunch of tutorials/examples of this on the internet.
import numpy as np
import random
import pandas
from sklearn.model_selection import train_test_split
class NeuralNetwork:
def __init__(self, layer_sizes):
self.layer_sizes = layer_sizes
weight_shapes = [(a, b) for a, b in zip(layer_sizes[1:], layer_sizes[:-1])]
self.weights = [
np.random.standard_normal(s) / s[1] ** 0.5 for s in weight_shapes
]
self.biases = [np.random.rand(s, 1) for s in layer_sizes[1:]]
def predict(self, a):
for w, b in zip(self.weights, self.biases):
a = self.activation(np.matmul(w, a) + b)
return a
def get_classify_score(self, images, labels):
predictions = self.predict(images)
num_correct = sum(
[np.argmax(a) == np.argmax(b) for a, b in zip(predictions, labels)]
)
return num_correct
def get_score(self, images, labels):
predictions = self.predict(images)
predictions = predictions.reshape(predictions.shape[0:2])
return -np.sum(np.abs(np.linalg.norm(predictions-labels)))
def clone(self):
nn = NeuralNetwork(self.layer_sizes)
nn.weights = np.copy(self.weights)
nn.biases = np.copy(self.biases)
return nn
def mutate(self):
for _ in range(self.weighted_random([(20, 1), (3, 2), (2, 3), (1, 4)])):
l = self.weighted_random([(l.flatten().shape[0], i) for i, l in enumerate(self.weights)])
shape = self.weights[l].shape
layer = self.weights[l].flatten()
layer[np.random.randint(0, layer.shape[0]-1)] = np.random.uniform(-2, 2)
self.weights[l] = layer.reshape(shape)
if np.random.uniform() < 0.01:
b = self.weighted_random([(b.flatten().shape[0], i) for i, b in enumerate(self.biases)])
shape = self.biases[b].shape
bias = self.biases[b].flatten()
bias[np.random.randint(0, bias.shape[0]-1)] = np.random.uniform(-2, 2)
self.biases[b] = bias.reshape(shape)
@staticmethod
def activation(x):
return 1 / (1 + np.exp(-x))
@staticmethod
def weighted_random(pairs):
total = sum(pair[0] for pair in pairs)
r = np.random.randint(1, total)
for (weight, value) in pairs:
r -= weight
if r <= 0: return value