Answer to exam 2011-06-01

1

1d
2f
3c
4g
5b
6e
7i
8a



2
Threshold function. 
y=1 for sum >=theta
y=0 for sum <theta
advantage: very fast to compute, a lot of theory can be developed for binary logic units, note that thresholding is a non-linear operation (but you can only represent 2 values of course)

Linear activation function
y=sum-theta for sum>=theta
y=0 for sum <theta
advantage: fast to compute, a lot of theory has been developed for linear systems

Sigmoid activation function
y=f(sum) where f is a sigmoidal function like the logistic function or the tan
advantage: nonlinear activation functions provide the network to perform universal approximation, which is far more powerful than anything you get from linear unit networks.


3

Our desision surface (line) should go from (0,2) to (1,0) for instance. Below this we classify points as class "1" and above as class "0".
For bias=1 we have w=[-1, -0.5]


4
input=(1 1 1 1)    net= -0.2    out= -1     target= 1     dW= (0.2 0.2 0.2 0.2)
w = (0.5, -1.0, 0.8), b=0.3



5
You are less likely to get dead units in the multi-winner as with the single-winner.
Single-winner is much faster, so you can have time to do more training (if time is an issue). 
You can set them to a small random value. Better is to calculate the mean and std of the data and randomly place the units within the region given by the mean and std.


6
Principal components are the directions of largest variation in the data. If you know that this variation contains information about your system (and thus that is is not due to noise or fluctuations that are of no interest to you), you can use the principal components to describe (model) your data with few parameters (few components). It thus leads to a dimension reductioon of your data representation.

You can use Sanger's rule, Oja's rule or APEX.

Oja: dw=eta*y*(x^T - y^T*w)


7
The generalization capacity of ANNs are used to approximate the value function.
It is kind of supervised, but the signal is typically a scalar value indicating ``good'' or ``bad''.


8
BP performs a gradient descent. This procedure can get stuck in local minima. The local minima do not necessarily represent good enough solutions.


9
These appear in Hopfield networks. These are stable states that do not correspond to any learned pattern. They appear as a side effect of learning. They also depend on correlations between patterns. 


10
Start by testing any random pattern, e.g. [1  1  1  1  1]. Iterate until you get a stable output. This will be one of the stored patterns. The anti-pattern of this will also be a pattern. Examples are [-1  1  1  1  -1] and [1  -1  -1  -1  1].

11
RBF-units are local whereas sigmoids have infinite tails. This means they only respond to a limited range of inputs. When you do learning, this means that when you change weights, not all RBFs will be affected and thus that learning one pattern does not necessarily conflict with learning of another pattern.

RBFs are commonly used as a first layer to code/create a representation of the data.


12

input:
T_air_t0 
T_water_t0 
sun_H_t0
T_air_t-1 
T_water_t-1 
sun_H_t-1
T_air_t-2 
T_water_t-2 
sun_H_t-2
T_air_t-3 
T_water_t-3 
sun_H_t-3
T_air_t-4 
T_water_t-4 
sun_H_t-4
T_air_t-5 
T_water_t-5 
sun_H_t-5

output:
T_water_t+1

Use a multi-layer perceptron trained with the BP algorithm. 
Start with 3 hidden nodes, increase and compare performance.
Segment the data using cross validation: a large set for training, a smaller set for test, and a smaller set for end test. Use data from 6 days (today and 5 days back) to predict tomorrow (use the data for that 7th day as correct answer). 


13
The problem is overlearning, e.g. too many hidden units were added. The problem comes from only looking at the error on the training set. You need to look at the error of the (unseen) test set. Split the data using cross validation (see Q3). This will lead to a final net with fewer hidden nodes, larger error on the training set, but smaller error on the test set.

14
If you have 256 elements=nodes, use the rule of Hopfield storage capacity. 
capacity~0.13N=0.13*256=33
You can hope to store and reliably retrieve correct answer for up to 33 persons.

(The above arguments hold for a data representation of equal number of 1s and 0s, e.g. a dense representation. If you use a sparse representation the capacity will scale up to O N*log(N)~616.)