Cluster Analysis
When does use Cluster analysis
1. To figure out what to predict
2. to detect patter of interest in data- when does not know enough what patter to expect
3. As an exploratory tool to understand the data
What is cluster analsys
PCA = Reduce number of cloumns
Cluster analysis = Reduce number of rows
Cluster Analysis:
1. Group objects(typically, rows or records or observations) in a data set based on similarity of the properties or attributes(columns) of the objects(rows)
2. There are at least a few hundred(if not more) approaches for performing cluster analysis
Pupular approaches to cluster Analysis
1. Hierarchical clustering(agglomerative) such as single Linkage, Average Linkage, and Complete linkage clustering
2. Partitioning clustering methods such as K-means and K-Medians for numeric data and K-Models for cluster categorical data
3. Overlapping clustering methods
4. Latent Class methods
K-means cluster
While performing partitioning(K-Means) and overlapping clustering (overlapping k-centroids)
1. Normalize normalize normalize
2. Severe local optima: Run procedure from at least 50 random starts for clustering into K- Groups. Otherwise, there is high risk of sub-optima cluster or groups
3. How many cluster to choose
a. USe Scree plot and interpret-ability
b. Don't Link solution don't use it
c. Unless a population is perfectly multi-model i.e. has exactly K models, once can extract as many cluster as one wants
4. Decide how you would handle missing data- missing values in some variables
a. Replace missing values in a cloumn by the mean of the non-missing values of each columns
b. Ignore row entirely if column has missing values
c. Imput the missing values by predicting modeling techniques
Case study
Assume you are a business analyst in a large finance company operating in the secondary (financial derivatives) market. You have access to their monthly operations database, with 500,000 trades and 200 variables, and are asked to summarize the key patterns and relationships in the data. How would you proceed?
above is centered around the Cluster Analysis topic. I would rather focus to explain the cluster analysis and how K-means clustering can be useful here
Stated problem can be covered under Unsupervized-Classification problem and we can use K-means clustering technique to solve the problem
First let me try to explain the clustering and when does it will be useful.
- To figure out what to predict
- To detect pattern of interest in data when we don have enough what patter to expect
- This can be used as exploratory tool to udernstand the data
Based on the available datasets(500000 trade data with 200 variables), we can use cluster analysis to identify key pattern and relationship in data
and to do the same, here is my approach to procceed
- We will first check the distribution of data if its normal or not
- Using the boxplot will try to detect outlier
- Check if there any outlier or missing values in data
- Treat the outlier/missing values by imputing and/or capping/florring method
- Whole concept is try to normalize data as much as possible
- plot the data using plot() or scree() plot in r and try to identify the possoble group
- Us K-mean cluster create cluster
- Based on the created cluster/group.
- find the mean values of Pricipal components of each group
- Draw the conclustion about the each group for business purpose
When does use Cluster analysis
1. To figure out what to predict
2. to detect patter of interest in data- when does not know enough what patter to expect
3. As an exploratory tool to understand the data
What is cluster analsys
PCA = Reduce number of cloumns
Cluster analysis = Reduce number of rows
Cluster Analysis:
1. Group objects(typically, rows or records or observations) in a data set based on similarity of the properties or attributes(columns) of the objects(rows)
2. There are at least a few hundred(if not more) approaches for performing cluster analysis
Pupular approaches to cluster Analysis
1. Hierarchical clustering(agglomerative) such as single Linkage, Average Linkage, and Complete linkage clustering
2. Partitioning clustering methods such as K-means and K-Medians for numeric data and K-Models for cluster categorical data
3. Overlapping clustering methods
4. Latent Class methods
K-means cluster
While performing partitioning(K-Means) and overlapping clustering (overlapping k-centroids)
1. Normalize normalize normalize
2. Severe local optima: Run procedure from at least 50 random starts for clustering into K- Groups. Otherwise, there is high risk of sub-optima cluster or groups
3. How many cluster to choose
a. USe Scree plot and interpret-ability
b. Don't Link solution don't use it
c. Unless a population is perfectly multi-model i.e. has exactly K models, once can extract as many cluster as one wants
4. Decide how you would handle missing data- missing values in some variables
a. Replace missing values in a cloumn by the mean of the non-missing values of each columns
b. Ignore row entirely if column has missing values
c. Imput the missing values by predicting modeling techniques
Case study
Assume you are a business analyst in a large finance company operating in the secondary (financial derivatives) market. You have access to their monthly operations database, with 500,000 trades and 200 variables, and are asked to summarize the key patterns and relationships in the data. How would you proceed?
above is centered around the Cluster Analysis topic. I would rather focus to explain the cluster analysis and how K-means clustering can be useful here
Stated problem can be covered under Unsupervized-Classification problem and we can use K-means clustering technique to solve the problem
First let me try to explain the clustering and when does it will be useful.
- To figure out what to predict
- To detect pattern of interest in data when we don have enough what patter to expect
- This can be used as exploratory tool to udernstand the data
Based on the available datasets(500000 trade data with 200 variables), we can use cluster analysis to identify key pattern and relationship in data
and to do the same, here is my approach to procceed
- We will first check the distribution of data if its normal or not
- Using the boxplot will try to detect outlier
- Check if there any outlier or missing values in data
- Treat the outlier/missing values by imputing and/or capping/florring method
- Whole concept is try to normalize data as much as possible
- plot the data using plot() or scree() plot in r and try to identify the possoble group
- Us K-mean cluster create cluster
- Based on the created cluster/group.
- find the mean values of Pricipal components of each group
- Draw the conclustion about the each group for business purpose
