Learning Vector Quantization (LVQ) and How to Determine The Initial Weights

Learning Vector Quantization (LVQ) and How to Determine The Initial Weights

        Learning Vector Quantization (LVQ) is one of the models of Artificial Neural Networks that uses competitive learning algorithms and is supervised. LVQ is a machine learning technique that is widely used for classification problems. LVQ is often used for classifications that have a high degree of accuracy and speed in terms of processing. In addition, LVQ is also widely used because LVQ which is a type of Artificial Neural Network that is relatively easy to implement and is quite concise considering the parameters used in the learning process of the network are not many.

        The LVQ network architecture consists of 2 layers, namely the input and output layers. The input layer consists of n nodes that represent the number of input features or input feature vector dimensions while the output layer consists of m nodes that represent the number of classes. Each node in the output layer represents a certain class that has been known (supervised). The LVQ network architecture is shown in Figure 1. Each node in the input layer (Xi) is connected by weight (wij) to each node in the output layer (Yj), so that each output node will have n pieces of weight, according to the number of nodes in the input layer. These weights form a weight vector with dimension n which represents each class (node) on the output layer called a reference vector (codebook).

Figure 1. Learning Vector Quantization Architecture

        The classification process using LVQ consists of 2 main stages, namely the training process and the testing process. The training process is used to build a classification model using training data. The final result of the training process is a classification model in LVQ in the form of network weights in the architecture in Figure 1 denoted by w. The next stage is the testing process. The testing process is carried out to evaluate how well the performance of the classification model has been generated in the training process in conducting the classification. The performance measurement that is usually used in classification problems is accuracy. The higher the value of accuracy means the more accurate the resulting classification model can classify test data into the correct class. This series of stages, namely the training process and the testing process must be carried out sequentially to produce a good classification model. The two stages are also usually carried out repeatedly with different parameter values ​​to produce an accurate classification model. After the best classification model is obtained, then the model is used to classify any input into each class according to the issues raised.

        The classification process using LVQ consists of 2 main stages, namely the training process and the testing process. The training process is used to build a classification model using training data. The final result of the training process is a classification model in LVQ in the form of network weights in the architecture in Figure 1 denoted by w. The next stage is the testing process. The testing process is carried out to evaluate how well the performance of the classification model has been generated in the training process in conducting the classification. The performance measurement that is usually used in classification problems is accuracy. The higher the value of accuracy means the more accurate the resulting classification model can classify test data into the correct class. This series of stages, namely the training process and the testing process must be carried out sequentially to produce a good classification model. The two stages are also usually carried out repeatedly with different parameter values ​​to produce an accurate classification model. After the best classification model is obtained, then the model is used to classify any input into each class according to the issues raised.

        During the training process, the output node will be positioned to get closer to the decision surface by updating the weight vector values ​​according to the input data in the supervised training process. The weight vector will be updated close to the input vector if input X and weight w (reference vector) are in the same class, but the weight vector will be updated away from the input vector if input X and weight w are not in the same class. The final result of the training process is the network weights, which can also be called the classification model of LVQ, therefore the search for optimal weights is the key to the classification problem using LVQ. The search for optimal weights in LVQ is strongly influenced by determining / initializing initial weights. Initial weight initialization or in LVQ is also called the easiest and most commonly used reference vector is by selecting the first n input vectors representing each class in the training process, where n is the number of classes to be classified. Initialization of reference vectors (weights) in this way is very sensitive to the accuracy of the system because inaccuracies in its selection can result in poor system accuracy.

        Another alternative for determining reference vectors is to use clustering techniques. Clustering is the process of grouping data into groups that we specify at the beginning of the amount, but the groups do not have labels / names. The use of clustering in LVQ aims to obtain the focal point of data clusters. Data that is grouped through the clustering process is assumed to be divided according to the label / class of the data, so that the focal point of each group (cluster) is considered to represent the existing classes. As a representative of existing classes, then this center point will be used as the initial weight (reference vector) of each class. After initial weight is determined using clustering techniques, the training process and the LVQ testing process are carried out as usual. Clustering techniques that can be used to determine this initial weight include Self Organizing Map (SOM) and K-Means clustering.

        Another alternative that can be used to determine a better initial weight is to use optimization methods such as the Genetic Algorithm and Particle Swarm Optimization (PSO). There is little difference in determining the initial weight using clustering techniques and optimization methods. In determining the initial weight by clustering technique, the clustering process will stand alone in forming data groups (clusters) which later we will just take the center point as initial weights, while in determining the initial weights using the optimization method, an evaluation of fitness values ​​or cost value. The value that affects the evaluation of fitness value or cost value is usually the accuracy value. The accuracy value is obtained by performing steps such as the testing process in the LVQ stage. The weight with the best fitness value or cost value will be chosen as the initial weight. After the initial weight is determined, the training process and testing process in the LVQ stage will be carried out as usual. 

        Clustering and optimization methods are alternatives that can be used to determine a better initial weight (reference vector) in LVQ. If asked which one is better, it is necessary to do further research on this, because the implementation of the LVQ method in carrying out the classification along with the modifications made in it will very much depend on the problem taken and the data used in it.