Neural Network: Remove confusions

I was really in a dillema on how to design an ANN. I know enough theory (at least gained by browsing and everything) but all of them where how it functions and how the structure is done. But none on how can I specifically design a NN for a specific function. At last, I found a quite good post (tutorial) which is written is really simple language and may help you.

http://www.ai-junkie.com/ann/evolved/nnt1.html

Just some confusions that I have removed while reading this tutorial are:

When do we need a hidden layer?
We need a hidden layer whenever out output space is not linear (like the XOR function). Simple, eh!!

Just how many input nodes do I need?
Input nodes depend on the number of different data you have to feed to the neural net. If you have a matrix of 9 cells and need to feed the matrix to the NN then you will need 9 input nodes. But it is also possible to feed the input as binary combinations. If I want to feed a specific locaiton to the NN from a set of 64 locations then probably a input set of 6 nodes will be enough (2^6 = 64).

How can we get the output?
The output is provided by the output layer of neurons. At each neuron there is a activation function, which takes the weighted sum of inputs and produces an output. This output can be binary, in case of a threshold activation function, or it can be continuous, in case of a sigmoid function.

What is a Bias?
As we know, for a simple Neuron the input is the summation of the input values multiplied by the weight and the output depends on whether this summation is greater than the threshold or not. That is:
x1w1 + x2w2 + ...... + xnwn >= t
If we bring the threshold value on the LHS then the equation becomes:
x1w1 + x2w2 + ...... + xnwn - t >= 0
=>x1w1 + x2w2 + ...... + xnwn + (-1)t >= 0
So, as we can see we can treat t also as an input and its weight is always -1. In this way, the threshold value will also evolve with the network and we dont have to think about it. Another advantage is that even if all the inputs are zero, because of the bias value (t) the output will be something, it will not be zero forever.

How does the NN learn or adapt?
The learning is based on the difference between the actual output and the output of the NN. This error is fed to the layers to correct the weight of the input so that the new output of the node is better than the previous one.
Learning can be done by various methods. Two of them are backpropagation and Genetic algorithm. When using backpropagation the error amount is fed back to the layers of the NN so that the weights and the bias of that layer is modified accordingly.

How can we decide on the number of neurons in the layers?
This is what seemed the most perplexing thing while designing a NN. Its natural that the input and output would be directly proportional to the number of input and the number of output we need, but what about the hidden layer? How many hidden layers do we need and how many neurons per hidden layer?
Well, it seems that one hidded layer is what we need. Almost all the jobs can be done using one hidded layer. And in case of number of neurons, I dont know :) They say that you are going to build a feel of it. There are no hard and fast rule of this. Too few neurons will make your result scratchy. And too many will take longer time and rather fune tune to the training data too much. That means, networks with too many neurons will also adapt to the noise data so much that it will be difficul to generalize it. An excellent resource would be:

http://www.dtreg.com/mlfn.htm

ANN with Genetic Algorithm:
If ANN is used with GA then the learning process is done by GA. The initial weight is fed to the GA along with the fitness function of each member of population. The neural network will evaluate the output of the network and increate or decrease the fitness value of the cromosome accordingly. After some time a new generation is created and the GA is called. The GA will then decide according to the fitness value and generate a new generation. The new generation of population with the new weights are then again fed to the neural nework and the loop continues.

For a Good introduction please refer to:
http://www.ai-junkie.com/ann/evolved/nnt1.html

Genetic Algorithm: A very simple primer

Here I am going to post a very short intuitive description of Genetic algorithm while I am reviewing and refreshing my knowledge which I gathered in my bachelors fourth year.

Genetic algorithm actually mimics the nature by first selecting the best persons in the population, then reproducing (in AI term, its called crossover) children from the best parents, and then mutating the new generation (as per Darwin).

Population: So what is actually a population. A population is a set of possible solutions to the problem. Normally, we randomly pick some solutions, and call those a population (initial population) and then process each of them through the engine to get their fitness (are they good solutions?) and depending upon that we assign each of the population a fitness value.

Crossover: After we get all the fitnees values, we can choose the best persons from the population and then we do cross over just like DNA crossover in human. It can be of several types. The simplest is in the case of binary input. Say, there are two best persons A and B and they are represented

A: 01010101

B: 11110000

And the crossover rule is to cross after 4th position. So the reproduction would be:

x: 01010000

y: 11110101

Now, to be more precise, crossover is not always done between the best persons in the population. The best persons marely have the best chance to be participating in a crossover funciton. Normally we use a function called a Roulette Wheel. It works like following:

The total population is charted like a pie chart with the people having a slice of the pie proportional to his fitness score. And then randomly choose a position in the pie and take that person.

Crossover rate: This rate defines the probability that two members of the population will have crossover. This probability is set to around 0.7

Mutation: Mutation refers to randomly change some bits of the children for evolution. This rate of mutation can be very small (like 0.001) or quite large (say 0.07) depending upon the scenario. We randomly flip the bits in case of binary population.

Actual Working: So, how the genetic algorithm actually work? With the primer above, you should be able to understand the following:

Generate enough possible random solutions to the problem, create a population of all the solutions. Check the solutions for accuracy. Create rule for fitness. The rule is upto you and depends on the problem in hand. It can be as simple as proportional or inverse proportional to the actual solution and the random solution. Give a fitness value to each member of the population. Apply roullette wheel algorithm to choose person to apply cross over. Then, apply crossover to the picked members and generate a new generation of children by mutating the crossovered members. So, now we got a whole new set of evolved population. Iterate the whole procedure again until the fitness value comes to a certain threshold where we stop and accept that the current population is the best.

A very good primer can be found at: http://www.ai-junkie.com/ga/intro/gat1.html