Cloud Hosting

There exist publicly accessible data which describe the socio-economic characteristics of a geographic location. In Australia where I reside, the Government through the Australian Bureau of Statistics (ABS) collects and publishes individual and household data on a regular basis in respect of income, occupation, education, employment and housing at an area level. Some examples of the published data points include:

Whilst these data points appear to focus heavily on individual people, it reflects peoples access to material and social resources, and their ability to participate in society in a particular geographic area, ultimately informing the socio-economic advantage and disadvantage of this area.

Given these data points, is there a way to derive a score which ranks geographic areas from the most to the least advantaged?

The goal to derive a score may formulate this as a regression problem, where each data point or feature is used to predict a target variable, in this scenario, a numerical score. This requires the target variable to be available in some instances for training the predictive model.

However, as we dont have a target variable to start with, we may need to approach this problem in another way. For instance, under the assumption that each geographic areas is different from a socio-economic standpoint, can we aim to understand which data points help explain the most variations, thereby deriving a score based on a numerical combination of these data points.

We can do exactly that using a technique called the Principal Component Analysis (PCA), and this article demonstrates how!

ABS publishes data points indicating the socio-economic characteristics of a geographic area in the Data Download section of this webpage, under the Standardised Variable Proportions data cube[1]. These data points are published at the Statistical Area 1 (SA1) level, which is a digital boundary segregating Australia into areas of population of approximately 200800 people. This is a much more granular digital boundary compared to the Postcode (Zipcode) or the States digital boundary.

For the purpose of demonstration in this article, Ill be deriving a socio-economic score based on 14 out of the 44 published data points provided in Table 1 of the data source above (Ill explain why I select this subset later on). These are :

In this section, Ill be stepping through the Python code for deriving a socio-economic score for a SA1 region in Australia using PCA.

Ill start by loading in the required Python packages and the data.

### For dataframe operationsimport numpy as npimport pandas as pd

### For PCAfrom sklearn.decomposition import PCAfrom sklearn.preprocessing import StandardScaler

### For Visualizationimport matplotlib.pyplot as pltimport seaborn as sns

### For Validationfrom scipy.stats import pearsonr

file1 = 'data/standardised_variables_seifa_2021.xlsx'

### Reading from Table 1, from row 5 onwards, for column A to ATdata1 = pd.read_excel(file1, sheet_name = 'Table 1', header = 5,usecols = 'A:AT')

data1_dropna = data1.dropna()

An important cleaning step before performing PCA is to standardise each of the 14 data points (features) to a mean of 0 and standard deviation of 1. This is primarily to ensure the loadings assigned to each feature by PCA (think of them as indicators of how important a feature is) are comparable across features. Otherwise, more emphasis, or higher loading, may be given to a feature which is actually not significant or vice versa.

Note that the ABS data source quoted above already have the features standardised. That said, for an unstandardised data source:

### Take all but the first column which is merely a location indicatordata_final = data1_dropna.iloc[:,1:]

### Perform standardisation of datasc = StandardScaler()sc.fit(data_final)

### Standardised datadata_final = sc.transform(data_final)

With the standardised data, PCA can be performed in just a few lines of code:

pca = PCA()pca.fit_transform(data_final)

PCA aims to represent the underlying data by Principal Components (PC). The number of PCs provided in a PCA is equal to the number of standardised features in the data. In this instance, 14 PCs are returned.

Each PC is a linear combination of all the standardised features, only differentiated by its respective loadings of the standardised feature. For example, the image below shows the loadings assigned to the first and second PCs (PC1 and PC2) by feature.

With 14 PCs, the code below provides a visualization of how much variation each PC explains:

exp_var_pca = pca.explained_variance_ratio_plt.bar(range(1, len(exp_var_pca) + 1), exp_var_pca, alpha = 0.7,label = '% of Variation Explained',color = 'darkseagreen')

plt.ylabel('Explained Variation')plt.xlabel('Principal Component')plt.legend(loc = 'best')plt.show()

As illustrated in the output visualization below, Principal Component 1 (PC1) accounts for the largest proportion of variance in the original dataset, with each following PC explaining less of the variance. To be specific, PC1 explains circa. 35% of the variation within the data.

For the purpose of demonstration in this article, PC1 is chosen as the only PC for deriving the socio-economic score, for the following reasons:

### Using df_plot dataframe per Image 1

sns.heatmap(df_plot, annot = False, fmt = ".1f", cmap = 'summer') plt.show()

To obtain a score for each SA1, we simply multiply the standardised portion of each feature by its PC1 loading. This can be achieved by:

### Perform sum product of standardised feature and PC1 loadingpca.fit_transform(data_final)

### Reverse the sign of the sum product above to make output more interpretablepca_data_transformed = -1.0*pca.fit_transform(data_final)

### Convert to Pandas dataframe, and join raw score with SA1 columnpca1 = pd.DataFrame(pca_data_transformed[:,0], columns = ['Score_Raw'])score_SA1 = pd.concat([data1_dropna['SA1_2021'].reset_index(drop = True), pca1], axis = 1)

### Inspect the raw scorescore_SA1.head()

The higher the score, the more advantaged a SA1 is in terms its access to socio-economic resource.

How do we know the score we derived above was even remotely correct?

For context, the ABS actually published a socio-economic score called the Index of Economic Resource (IER), defined on the ABS website as:

The Index of Economic Resources (IER) focuses on the financial aspects of relative socio-economic advantage and disadvantage, by summarising variables related to income and housing. IER excludes education and occupation variables as they are not direct measures of economic resources. It also excludes assets such as savings or equities which, although relevant, cannot be included as they are not collected in the Census.

Without disclosing the detailed steps, the ABS stated in their Technical Paper that the IER was derived using the same features (14) and methodology (PCA, PC1 only) as what we had performed above. That is, if we did derive the correct scores, they should be comparable against the IER scored published here (Statistical Area Level 1, Indexes, SEIFA 2021.xlsx, Table 4).

As the published score is standardised to a mean of 1,000 and standard deviation of 100, we start the validation by standardising the raw score the same:

score_SA1['IER_recreated'] = (score_SA1['Score_Raw']/score_SA1['Score_Raw'].std())*100 + 1000

For comparison, we read in the published IER scores by SA1:

file2 = 'data/Statistical Area Level 1, Indexes, SEIFA 2021.xlsx'

data2 = pd.read_excel(file2, sheet_name = 'Table 4', header = 5,usecols = 'A:C')

data2.rename(columns = {'2021 Statistical Area Level 1 (SA1)': 'SA1_2021', 'Score': 'IER_2021'}, inplace = True)

col_select = ['SA1_2021', 'IER_2021']data2 = data2[col_select]

ABS_IER_dropna = data2.dropna().reset_index(drop = True)

Validation 1 PC1 Loadings

As shown in the image below, comparing the PC1 loading derived above against the PC1 loading published by the ABS suggests that they differ by a constant of -45%. As this is merely a scaling difference, it doesnt impact the derived scores which are standardised (to a mean of 1,000 and standard deviation of 100).

(You should be able to verify the Derived (A) column with the PC1 loadings in Image 1).

Validation 2 Distribution of Scores

The code below creates a histogram for both scores, whose shapes look to be almost identical.

score_SA1.hist(column = 'IER_recreated', bins = 100, color = 'darkseagreen')plt.title('Distribution of recreated IER scores')

ABS_IER_dropna.hist(column = 'IER_2021', bins = 100, color = 'lightskyblue')plt.title('Distribution of ABS IER scores')

plt.show()

Validation 3 IER score by SA1

As the ultimate validation, lets compare the IER scores by SA1:

## Plot scores on x-y axis. ## If scores are identical, it should show a straight line.

plt.scatter('IER_recreated', 'IER_2021', data = IER_join, color = 'darkseagreen')plt.title('Comparison of recreated and ABS IER scores')plt.xlabel('Recreated IER score')plt.ylabel('ABS IER score')

plt.show()

A diagonal straight line as shown in the output image below supports that the two scores are largely identical.

To add to this, the code below shows the two scores have a correlation close to 1:

The demonstration in this article effectively replicates how the ABS calibrates the IER, one of the four socio-economic indexes it publishes, which can be used to rank the socio-economic status of a geographic area.

Taking a step back, what weve achieved in essence is a reduction in dimension of the data from 14 to 1, losing some information conveyed by the data.

Dimensionality reduction technique such as the PCA is also commonly seen in helping to reduce high-dimension space such as text embeddings to 23 (visualizable) Principal Components.

Visit link:

Deriving a Score to Show Relative Socio-Economic Advantage and Disadvantage of a Geographic Area - Towards Data Science