Firstly, I'd suggest using something like PCA as a dimensionality reduction method, but if you have to roll your own then your question is insufficiently constrained. Where two columns are correlated, which one do you want to remove? What if column A is correlated with column B, while column B is correlated with column C, but not column A?

You can get a pairwise matrix of correlations by calling DataFrame.corr() (docs) which might help you with developing your algorithm, but eventually you need to convert that into a list of columns to keep.

Plug your features dataframe in this function and just set your correlation threshold. It'll auto drop columns, but will also give you a diagnostic of the columns it drops if you want to do it manually.

def corr_df(x, corr_val):
'''
Obj: Drops features that are strongly correlated to other features.
This lowers model complexity, and aids in generalizing the model.
Inputs:
df: features df (x)
corr_val: Columns are dropped relative to the corr_val input (e.g. 0.8)
Output: df that only includes uncorrelated features
'''


# Creates Correlation Matrix and Instantiates
corr_matrix = x.corr()
iters = range(len(corr_matrix.columns) - 1)
drop_cols = []


# Iterates through Correlation Matrix Table to find correlated columns
for i in iters:
for j in range(i):
item = corr_matrix.iloc[j:(j+1), (i+1):(i+2)]
col = item.columns
row = item.index
val = item.values
if val >= corr_val:
# Prints the correlated feature set and the corr val
print(col.values[0], "|", row.values[0], "|", round(val[0][0], 2))
drop_cols.append(i)


drops = sorted(set(drop_cols))[::-1]


# Drops the correlated columns
for i in drops:
col = x.iloc[:, (i+1):(i+2)].columns.values
df = x.drop(col, axis=1)


return df

Here is the approach which I have used -

def correlation(dataset, threshold):
col_corr = set() # Set of all the names of deleted columns
corr_matrix = dataset.corr()
for i in range(len(corr_matrix.columns)):
for j in range(i):
if (corr_matrix.iloc[i, j] >= threshold) and (corr_matrix.columns[j] not in col_corr):
colname = corr_matrix.columns[i] # getting the name of column
col_corr.add(colname)
if colname in dataset.columns:
del dataset[colname] # deleting the column from the dataset


print(dataset)

Hope this helps!

You can use the following for a given data frame df:

corr_matrix = df.corr().abs()
high_corr_var=np.where(corr_matrix>0.8)
high_corr_var=[(corr_matrix.columns[x],corr_matrix.columns[y]) for x,y in zip(*high_corr_var) if x!=y and x<y]

I took the liberty to modify TomDobbs' answer. The reported bug in the comments is removed now. Also, the new function filters out the negative correlation, too.

def corr_df(x, corr_val):
'''
Obj: Drops features that are strongly correlated to other features.
This lowers model complexity, and aids in generalizing the model.
Inputs:
df: features df (x)
corr_val: Columns are dropped relative to the corr_val input (e.g. 0.8)
Output: df that only includes uncorrelated features
'''


# Creates Correlation Matrix and Instantiates
corr_matrix = x.corr()
iters = range(len(corr_matrix.columns) - 1)
drop_cols = []


# Iterates through Correlation Matrix Table to find correlated columns
for i in iters:
for j in range(i):
item = corr_matrix.iloc[j:(j+1), (i+1):(i+2)]
col = item.columns
row = item.index
val = item.values
if abs(val) >= corr_val:
# Prints the correlated feature set and the corr val
print(col.values[0], "|", row.values[0], "|", round(val[0][0], 2))
drop_cols.append(i)


drops = sorted(set(drop_cols))[::-1]


# Drops the correlated columns
for i in drops:
col = x.iloc[:, (i+1):(i+2)].columns.values
x = x.drop(col, axis=1)
return x

The method here worked well for me, only a few lines of code: https://chrisalbon.com/machine_learning/feature_selection/drop_highly_correlated_features/

import numpy as np


# Create correlation matrix
corr_matrix = df.corr().abs()


# Select upper triangle of correlation matrix
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))


# Find features with correlation greater than 0.95
to_drop = [column for column in upper.columns if any(upper[column] > 0.95)]


# Drop features
df.drop(to_drop, axis=1, inplace=True)

A small revision to the solution posted by user3025698 that resolves an issue where the correlation between the first two columns is not captured and some data type checking.

def filter_df_corr(inp_data, corr_val):
'''
Returns an array or dataframe (based on type(inp_data) adjusted to drop \
columns with high correlation to one another. Takes second arg corr_val
that defines the cutoff


----------
inp_data : np.array, pd.DataFrame
Values to consider
corr_val : float
Value [0, 1] on which to base the correlation cutoff
'''
# Creates Correlation Matrix
if isinstance(inp_data, np.ndarray):
inp_data = pd.DataFrame(data=inp_data)
array_flag = True
else:
array_flag = False
corr_matrix = inp_data.corr()


# Iterates through Correlation Matrix Table to find correlated columns
drop_cols = []
n_cols = len(corr_matrix.columns)


for i in range(n_cols):
for k in range(i+1, n_cols):
val = corr_matrix.iloc[k, i]
col = corr_matrix.columns[i]
row = corr_matrix.index[k]
if abs(val) >= corr_val:
# Prints the correlated feature set and the corr val
print(col, "|", row, "|", round(val, 2))
drop_cols.append(col)


# Drops the correlated columns
drop_cols = set(drop_cols)
inp_data = inp_data.drop(columns=drop_cols)
# Return same type as inp
if array_flag:
return inp_data.values
else:
return inp_data

This is the approach I used on my job last month. Perhaps it is not the best or quickest way, but it works fine. Here, df is my original Pandas dataframe:

dropvars = []
threshold = 0.95
df_corr = df.corr().stack().reset_index().rename(columns={'level_0': 'Var 1', 'level_1': 'Var 2', 0: 'Corr'})
df_corr = df_corr[(df_corr['Corr'].abs() >= threshold) & (df_corr['Var 1'] != df_corr['Var 2'])]
while len(df_corr) > 0:
var = df_corr['Var 1'].iloc[0]
df_corr = df_corr[((df_corr['Var 1'] != var) & (df_corr['Var 2'] != var))]
dropvars.append(var)
df.drop(columns=dropvars, inplace=True)

My idea is as follows: first, I create a dataframe containing columna Var 1, Var 2 and Corr, where I keep only those pairs of variables whose correlation is higher than or equal my threshold (in absolute value). Then, I iteratively choose the first variable (Var 1 value) in this correlations dataframe, add it to dropvar list, and remove all lines of the correlations dataframe where it appears, until my correlations dataframe is empty. In the end, I remove the columns in my dropvar list from my original dataframe.

I had a similar question today and came across this post. This is what I ended up with.

def uncorrelated_features(df, threshold=0.7):
"""
Returns a subset of df columns with Pearson correlations
below threshold.
"""


corr = df.corr().abs()
keep = []
for i in range(len(corr.iloc[:,0])):
above = corr.iloc[:i,i]
if len(keep) > 0: above = above[keep]
if len(above[above < threshold]) == len(above):
keep.append(corr.columns.values[i])


return df[keep]

You can test this code below ?

Load libraries import

  pandas as pd
import numpy as np
# Create feature matrix with two highly correlated features


X = np.array([[1, 1, 1],
[2, 2, 0],
[3, 3, 1],
[4, 4, 0],
[5, 5, 1],
[6, 6, 0],
[7, 7, 1],
[8, 7, 0],
[9, 7, 1]])


# Convert feature matrix into DataFrame
df = pd.DataFrame(X)


# View the data frame
df


# Create correlation matrix
corr_matrix = df.corr().abs()


# Select upper triangle of correlation matrix
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))


# Find index of feature columns with correlation greater than 0.95
to_drop = [column for column in upper.columns if any(upper[column] > 0.95)]
# Drop features
df.drop(df[to_drop], axis=1)

I write my own way without any for loop to delete high covariance data from pandas dataframe

#get co variance of data
coVar = df.corr() # or df.corr().abs()
threshold = 0.5 #
"""
1. .where(coVar != 1.0) set NaN where col and index is 1
2. .where(coVar >= threshold) if not greater than threshold set Nan
3. .fillna(0) Fill NaN with 0
4. .sum() convert data frame to serise with sum() and just where is co var greater than threshold sum it
5. > 0 convert all Series to Boolean
"""


coVarCols = coVar.where(coVar != 1.0).where(coVar >=threshold).fillna(0).sum() > 0


# Not Boolean Becuase we need to delete where is co var greater than threshold
coVarCols = ~coVarCols


# get where you want
df[coVarCols[coVarCols].index]

I hope that's can help to use own pandas function to work with out any for loop, That's can help Improve your speed in big dataset

I found the answer provided by TomDobbs quite useful, however it doesn't work as intended. It has two problems:

• it misses the last pair of variables in each of correlation matrix rows/columns.
• it fails to remove one of each pair of collinear variables from the returned dataframe.

My revised version below corrects these issues:

def remove_collinear_features(x, threshold):
'''
Objective:
Remove collinear features in a dataframe with a correlation coefficient
greater than the threshold. Removing collinear features can help a model
to generalize and improves the interpretability of the model.


Inputs:
x: features dataframe
threshold: features with correlations greater than this value are removed


Output:
dataframe that contains only the non-highly-collinear features
'''


# Calculate the correlation matrix
corr_matrix = x.corr()
iters = range(len(corr_matrix.columns) - 1)
drop_cols = []


# Iterate through the correlation matrix and compare correlations
for i in iters:
for j in range(i+1):
item = corr_matrix.iloc[j:(j+1), (i+1):(i+2)]
col = item.columns
row = item.index
val = abs(item.values)


# If correlation exceeds the threshold
if val >= threshold:
# Print the correlated features and the correlation value
print(col.values[0], "|", row.values[0], "|", round(val[0][0], 2))
drop_cols.append(col.values[0])


# Drop one of each pair of correlated columns
drops = set(drop_cols)
x = x.drop(columns=drops)


return x

If you run out of memory due to pandas .corr() you may find the following solution useful:

    import numpy as np
from numba import jit
    

@jit(nopython=True)
def corr_filter(X, threshold):
n = X.shape[1]
columns = np.ones((n,))
for i in range(n-1):
for j in range(i+1, n):
if columns[j] == 1:
correlation = np.abs(np.corrcoef(X[:,i], X[:,j])[0,1])
if correlation >= threshold:
columns[j] = 0
return columns
    

columns = corr_filter(df.values, 0.7).astype(bool)
selected_columns = df.columns[columns]

correlatedColumns = []
corr = df.corr()
indices = corr.index
columns = corr.columns
posthreshold = 0.7
negthreshold = -0.7


for c in columns:
for r in indices:
if c != r and (corr[c][r] > posthreshold or corr[c][r] < negthreshold):
correlatedColumns.append({"column" : c , "row" : r , "val" :corr[c][r] })
            



print(correlatedColumns)

Here is an Auto ML class I created to eliminate multicollinearity between features.

What makes my code unique is that out two features that have high correlation, I have eliminated the feature that is least correlated with the target! I got the idea from this seminar by Vishal Patel Sir - https://www.youtube.com/watch?v=ioXKxulmwVQ&feature=youtu.be

#Feature selection class to eliminate multicollinearity
class MultiCollinearityEliminator():
    

#Class Constructor
def __init__(self, df, target, threshold):
self.df = df
self.target = target
self.threshold = threshold


#Method to create and return the feature correlation matrix dataframe
def createCorrMatrix(self, include_target = False):
#Checking we should include the target in the correlation matrix
if (include_target == False):
df_temp = self.df.drop([self.target], axis =1)
            

#Setting method to Pearson to prevent issues in case the default method for df.corr() gets changed
#Setting min_period to 30 for the sample size to be statistically significant (normal) according to
#central limit theorem
corrMatrix = df_temp.corr(method='pearson', min_periods=30).abs()
#Target is included for creating the series of feature to target correlation - Please refer the notes under the
#print statement to understand why we create the series of feature to target correlation
elif (include_target == True):
corrMatrix = self.df.corr(method='pearson', min_periods=30).abs()
return corrMatrix


#Method to create and return the feature to target correlation matrix dataframe
def createCorrMatrixWithTarget(self):
#After obtaining the list of correlated features, this method will help to view which variables
#(in the list of correlated features) are least correlated with the target
#This way, out the list of correlated features, we can ensure to elimate the feature that is
#least correlated with the target
#This not only helps to sustain the predictive power of the model but also helps in reducing model complexity
        

#Obtaining the correlation matrix of the dataframe (along with the target)
corrMatrix = self.createCorrMatrix(include_target = True)
#Creating the required dataframe, then dropping the target row
#and sorting by the value of correlation with target (in asceding order)
corrWithTarget = pd.DataFrame(corrMatrix.loc[:,self.target]).drop([self.target], axis = 0).sort_values(by = self.target)
print(corrWithTarget, '\n')
return corrWithTarget


#Method to create and return the list of correlated features
def createCorrelatedFeaturesList(self):
#Obtaining the correlation matrix of the dataframe (without the target)
corrMatrix = self.createCorrMatrix(include_target = False)
colCorr = []
#Iterating through the columns of the correlation matrix dataframe
for column in corrMatrix.columns:
#Iterating through the values (row wise) of the correlation matrix dataframe
for idx, row in corrMatrix.iterrows():
if(row[column]>self.threshold) and (row[column]<1):
#Adding the features that are not already in the list of correlated features
if (idx not in colCorr):
colCorr.append(idx)
if (column not in colCorr):
colCorr.append(column)
print(colCorr, '\n')
return colCorr


#Method to eliminate the least important features from the list of correlated features
def deleteFeatures(self, colCorr):
#Obtaining the feature to target correlation matrix dataframe
corrWithTarget = self.createCorrMatrixWithTarget()
for idx, row in corrWithTarget.iterrows():
print(idx, '\n')
if (idx in colCorr):
self.df = self.df.drop(idx, axis =1)
break
return self.df


#Method to run automatically eliminate multicollinearity
def autoEliminateMulticollinearity(self):
#Obtaining the list of correlated features
colCorr = self.createCorrelatedFeaturesList()
while colCorr != []:
#Obtaining the dataframe after deleting the feature (from the list of correlated features)
#that is least correlated with the taregt
self.df = self.deleteFeatures(colCorr)
#Obtaining the list of correlated features
colCorr = self.createCorrelatedFeaturesList()
return self.df

At first, thanks to TomDobbs and Synergix for their code. Below I am sharing my modifield version with some additions:

1. Between two correlated variables this function drops a variable which has the least correlation with the target variable
2. Added some useful logs (set verbose to True for log printing)

def remove_collinear_features(df_model, target_var, threshold, verbose):
'''
Objective:
Remove collinear features in a dataframe with a correlation coefficient
greater than the threshold and which have the least correlation with the target (dependent) variable. Removing collinear features can help a model
to generalize and improves the interpretability of the model.


Inputs:
df_model: features dataframe
target_var: target (dependent) variable
threshold: features with correlations greater than this value are removed
verbose: set to "True" for the log printing


Output:
dataframe that contains only the non-highly-collinear features
'''


# Calculate the correlation matrix
corr_matrix = df_model.drop(target_var, 1).corr()
iters = range(len(corr_matrix.columns) - 1)
drop_cols = []
dropped_feature = ""


# Iterate through the correlation matrix and compare correlations
for i in iters:
for j in range(i+1):
item = corr_matrix.iloc[j:(j+1), (i+1):(i+2)]
col = item.columns
row = item.index
val = abs(item.values)


# If correlation exceeds the threshold
if val >= threshold:
# Print the correlated features and the correlation value
if verbose:
print(col.values[0], "|", row.values[0], "|", round(val[0][0], 2))
col_value_corr = df_model[col.values[0]].corr(df_model[target_var])
row_value_corr = df_model[row.values[0]].corr(df_model[target_var])
if verbose:
print("{}: {}".format(col.values[0], np.round(col_value_corr, 3)))
print("{}: {}".format(row.values[0], np.round(row_value_corr, 3)))
if col_value_corr < row_value_corr:
drop_cols.append(col.values[0])
dropped_feature = "dropped: " + col.values[0]
else:
drop_cols.append(row.values[0])
dropped_feature = "dropped: " + row.values[0]
if verbose:
print(dropped_feature)
print("-----------------------------------------------------------------------------")


# Drop one of each pair of correlated columns
drops = set(drop_cols)
df_model = df_model.drop(columns=drops)


print("dropped columns: ")
print(list(drops))
print("-----------------------------------------------------------------------------")
print("used columns: ")
print(df_model.columns.tolist())


return df_model

The question here refers to a HUGE dataset. However, all of the answers I see are dealing with dataframes. I present an answer for a scipy sparse matrix which runs in parallel. Rather than returning a giant correlation matrix, this returns a feature mask of fields to keep after checking all fields for both positive and negative Pearson correlations.

I also try to minimize calculations using the following strategy:

• Process each column
• Start at the current column + 1 and calculate correlations moving to the right.
• For any abs(correlation) >= threshold, mark the current column for removal and calculate no further correlations.
• Perform these steps for each column in the dataset except the last.

This might be sped up further by keeping a global list of columns marked for removal and skipping further correlation calculations for such columns, since columns will execute out of order. However, I do not know enough about race conditions in python to implement this tonight.

Returning a column mask will obviously allow the code to handle much larger datasets than returning the entire correlation matrix.

Check each column using this function:

def get_corr_row(idx_num, sp_mat, thresh):
# slice the column at idx_num
cols = sp_mat.shape[1]
x = sp_mat[:,idx_num].toarray().ravel()
start = idx_num + 1
    

# Now slice each column to the right of idx_num
for i in range(start, cols):
y = sp_mat[:,i].toarray().ravel()
# Check the pearson correlation
corr, pVal = pearsonr(x,y)
# Pearson ranges from -1 to 1.
# We check both positive and negative correlations >= thresh using abs(corr)
if abs(corr) >= thresh:
# stop checking after finding the 1st correlation > thresh
return False
# Mark column at idx_num for removal in the mask
return True
    

Run the column level correlation checks in parallel:

from joblib import Parallel, delayed
import multiprocessing




def Get_Corr_Mask(sp_mat, thresh, n_jobs=-1):
    

# we must make sure the matrix is in csc format
# before we start doing all these column slices!
sp_mat = sp_mat.tocsc()
cols = sp_mat.shape[1]
    

if n_jobs == -1:
# Process the work on all available CPU cores
num_cores = multiprocessing.cpu_count()
else:
# Process the work on the specified number of CPU cores
num_cores = n_jobs


# Return a mask of all columns to keep by calling get_corr_row()
# once for each column in the matrix
return Parallel(n_jobs=num_cores, verbose=5)(delayed(get_corr_row)(i, sp_mat, thresh)for i in range(cols))

General Usage:

#Get the mask using your sparse matrix and threshold.
corr_mask = Get_Corr_Mask(X_t_fpr, 0.95)


# Remove features that are >= 95% correlated
X_t_fpr_corr = X_t_fpr[:,corr_mask]

in my code i need to remove low correlated columns with the dependent variable, and i got this code

to_drop = pd.DataFrame(to_drop).fillna(True)
to_drop = list(to_drop[to_drop['SalePrice'] <.4 ].index)
df_h1.drop(to_drop,axis=1)

df_h1 is my dataframe and SalePrice is the dependent variable... i think changing the value may suit for all other problems

The below snippet drop the most correlated features recursively.

def get_corr_feature(df):
corr_matrix = df.corr().abs()
# Select upper triangle of correlation matrix
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool_))
upper['score']= upper.max(axis=1)
upper.sort_values(by=['score'],ascending=False)
#Find the most correlated feature and send return it for drop
column_name=upper.sort_values(by=['score'],ascending=False).index[0]
max_score=upper.loc[column_name,'score']
return column_name, max_score


max_score=1
while max_score>0.5:
column_name, max_score=get_corr_feature(df)
df.drop(column_name,axis=1,inplace=True)

I wrote a notebook that uses partial correlations

https://gist.github.com/thistleknot/ce1fc38ea9fcb1a8dafcfe6e0d8af475

the gist of it (pun intended)

for train_index, test_index in kfold.split(all_data):
#print(iteration)
max_pvalue = 1
    

subset = all_data.iloc[train_index].loc[:, ~all_data.columns.isin([exclude])]
    

#skip y and states
set_ = subset.loc[:, ~subset.columns.isin([target])].columns.tolist()
    

n=len(subset)
    

while(max_pvalue>=.05):


dist = scipy.stats.beta(n/2 - 1, n/2 - 1, loc=-1, scale=2)
p_values = pd.DataFrame(2*dist.cdf(-abs(subset.pcorr()[target]))).T
p_values.columns = list(subset.columns)
        

max_pname = p_values.idxmax(axis=1)[0]
max_pvalue = p_values[max_pname].values[0]
        

if (max_pvalue > .05):


set_.remove(max_pname)
temp = [target]
temp.extend(set_)
subset = subset[temp]
    

winners = p_values.loc[:, ~p_values.columns.isin([target])].columns.tolist()
sig_table = (sig_table + np.where(all_data.columns.isin(winners),1,0)).copy()
    

signs_table[all_data.columns.get_indexer(winners)]+=np.where(subset.pcorr()[target][winners]<0,-1,1)




significance = pd.DataFrame(sig_table).T
significance.columns = list(all_data.columns)
display(significance)


sign = pd.DataFrame(signs_table).T
sign.columns = list(all_data.columns)
display(sign)


purity = abs((sign/num_folds)*(sign/significance)).T.replace([np.inf, -np.inf, np.NaN], 0)
display(purity.T)

I know that there are already a lot of answers on that but one way I found very simple and short is the following:

# Get correlation matrix
corr = X.corr()


# Create a mask for values above 90%
# But also below 100% since it variables correlated with the same one
mask = (X.corr() > 0.9) & (X.corr() < 1.0)
high_corr = corr[mask]


# Create a new column mask using any() and ~
col_to_filter_out = ~high_corr[mask].any()


# Apply new mask
X_clean = X[high_corr.columns[col_to_filter_out]]


# Visualize cleaned dataset
X_clean

I believe this has to be done in an iterative way:

uncorrelated_features = features.copy()


# Loop until there's nothing to drop
while True:
# Calculating the correlation matrix for the remaining list of features
cor = uncorrelated_features.corr().abs()


# Generating a square matrix with all 1s except for the main axis
zero_main = np.triu(np.ones(cor.shape), k=1) +
np.tril(np.ones(cor.shape), k=-1)


# Using the zero_main matrix to filter out the main axis of the correlation matrix
except_main = cor.where(zero_main.astype(bool))


# Calculating some metrics for each column, including the max correlation,
# mean correlation and the name of the column
mertics = [(except_main[column].max(), except_main[column].mean(), column) for column in except_main.columns]


# Sort the list to find the most suitable candidate to drop at index 0
mertics.sort(key=lambda x: (x[0], x[1]), reverse=True)


# Check and see if there's anything to drop from the list of features
if mertics[0][0] > 0.5:
uncorrelated_features.drop(mertics[0][2], axis=1, inplace=True)
else:
break

It's worth mentioning that you might want to customize the way I sorted the metrics list and/or how I detected whether I want to drop the column or not.

If you wanted to return a breakdown of correlated columns you could use this function to look at them to see what you are dropping and adjust your threshold

def corr_cols(df,thresh):
# Create correlation matrix
corr_matrix = df.corr().abs()
# Select upper triangle of correlation matrix
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool_))


dic = {'Feature_1':[],'Featur_2':[],'val':[]}
for col in upper.columns:
corl = list(filter(lambda x: x >= thresh, upper[col] ))
#print(corl)
if len(corl) > 0:
inds = [round(x,4) for x in corl]
for ind in inds:
#print(col)
#print(ind)
col2 = upper[col].index[list(upper[col].apply(lambda x: round(x,4))).index(ind)]
#print(col2)
dic['Feature_1'].append(col)
dic['Featur_2'].append(col2)
dic['val'].append(ind)
return pd.DataFrame(dic).sort_values(by="val", ascending=False)

And then remove them by calling the df

    corr = corr_cols(star,0.5)
df.drop(columns = corr.iloc[:,0].unique())

I manage to do it using this way. Kindly have a try. However, the way I did is just reached display purposes as I want to capture the result in my report. If you want to drop it, you can choose any columns from the dataframe below to drop it since can just choose either 1.

row_index = 0
corrDict = {}
row_name = []
col_name = []
corr_val = []


while row_index < len(df.corr().index.tolist()):
for index, x in enumerate(df.corr().iloc[row_index, :]):
if abs(x) >= 0.8 and index != row_index:
if abs(x) in corr_val:
if (df.corr().index.tolist()[row_index] in col_name) and (df.corr().columns.tolist()[index] in row_name):
continue
row_name.append(df.corr().index.tolist()[row_index])
col_name.append(df.corr().columns.tolist()[index])
corr_val.append(x)
row_index += 1
    

corrDict ={"First Feature (FF)": row_name, "Second Feature (SF)": col_name, "Correlation (FF x SF)": corr_val}
corr_df2=pd.DataFrame(corrDict)
corr_df2

This is my output:

You can choose either First Feature (FF) or Second Feature (SF). To drop highly correlated features from your original dataset:
your_df.drop(corr_df2['First Feature (FF)'].tolist(), axis=1, inplace=True)

There are three challenges to this problem. First, if features x and y are correlated, you don't want to use an algorithm that drops both. Second, if x and y are pairwise correlated and features y and z are also pairwise correlated, you want the algorithm to only remove y. In this sense, you want it to remove the minimum number of features so that no remaining features have correlations above your threshold. Third, from an efficiency standpoint, you do not want to have to compute the correlation matrix more than once.

Here's an option:

def corr_cleaner(df,corr_cutoff):
'''
df: pandas dataframe with column headers.
corr_cutoff: float between 0 and 1.
'''
abs_corr_matrix = df.corr().abs()
filtered_cols = []
while True:
offenders = []
for i in range(len(abs_corr_matrix)):
for j in range(len(abs_corr_matrix)):
if i != j:
if abs_corr_matrix.iloc[i,j] > corr_cutoff:
offenders.append(df.columns[i])


if len(offenders) > 0: # if at least one high correlation remains
c = Counter(offenders)
worst_offender = c.most_common(1)[0][0]  # var name of worst offender
del df[worst_offender]
filtered_cols.append(worst_offender)
abs_corr_matrix.drop(worst_offender, axis=0, inplace=True) #drop from x-axis
abs_corr_matrix.drop(worst_offender, axis=1, inplace=True) #drop from y-axis
else: # if no high correlations remain, break
break


return df, filtered_cols