Skip to main content

Neural Networks Fundamentals

Neural Networks are computing systems inspired by biological neural networks. They consist of interconnected nodes (neurons) that process information.

Basic Structure​

Perceptron​

The simplest form of a neural network:

import numpy as np

class Perceptron:
    def __init__(self, learning_rate=0.01, n_iterations=1000):
        self.learning_rate = learning_rate
        self.n_iterations = n_iterations

    def fit(self, X, y):
        # Initialize weights
        self.weights = np.zeros(1 + X.shape[1])

        for _ in range(self.n_iterations):
            for xi, target in zip(X, y):
                update = self.learning_rate * (target - self.predict(xi))
                self.weights[1:] += update * xi
                self.weights[0] += update

    def net_input(self, X):
        return np.dot(X, self.weights[1:]) + self.weights[0]

    def predict(self, X):
        return np.where(self.net_input(X) >= 0.0, 1, -1)

Multi-Layer Perceptron (MLP)​

Architecture​

  • Input Layer: Receives input features
  • Hidden Layer(s): Process information
  • Output Layer: Produces final output

Activation Functions​

Common activation functions:

import numpy as np
import matplotlib.pyplot as plt

# Sigmoid
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# ReLU
def relu(x):
    return np.maximum(0, x)

# Tanh
def tanh(x):
    return np.tanh(x)

# Plotting
x = np.linspace(-5, 5, 100)
plt.figure(figsize=(12, 4))

plt.subplot(1, 3, 1)
plt.plot(x, sigmoid(x))
plt.title('Sigmoid')

plt.subplot(1, 3, 2)
plt.plot(x, relu(x))
plt.title('ReLU')

plt.subplot(1, 3, 3)
plt.plot(x, tanh(x))
plt.title('Tanh')

plt.tight_layout()
plt.show()

Backpropagation​

The algorithm used to train neural networks:

  1. Forward Pass: Calculate output
  2. Calculate Loss: Compare with target
  3. Backward Pass: Calculate gradients
  4. Update Weights: Adjust based on gradients
  • TensorFlow: Google's open-source library
  • PyTorch: Facebook's research-focused framework
  • Keras: High-level API for TensorFlow

Simple Neural Network with Keras​

import tensorflow as tf
from tensorflow import keras

# Create model
model = keras.Sequential([
    keras.layers.Dense(128, activation='relu', input_shape=(784,)),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(10, activation='softmax')
])

# Compile model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train model
# model.fit(x_train, y_train, epochs=5)