What is Synergy Effect in Linear Regression | Machine Learning
Synergy Effect or Interaction Effect is a phenomenon that arises in the multiple linear regression setting in machine learning, when increase in the value of one Independent variable increases the impact of another Independent variable on the dependent Variable.
It’s okay if this above statement is not easily understandable. Let’s look at an example
Note: You should be familiar with multiple linear regression to understand this
Advertisement Sales Data-set
Let’s consider a data-set with advertising budget of a company for different categories and the sales of the company. So, the 3 columns of the data-set are Radio Advertising, Newspaper Advertising and Sales.
Let’s Fit a linear model in this data-set. This linear model’s equation would look like this.
Y = beta0 + beta1*radio + beta2* newspaper
Where beta0 , beta1 and beta2 are the weights and biases to be learnt.
Even though this might work fine, There are some cases where this model’s predictions might underestimate the actual sales in certain areas and overestimate the actual sales in other areas.
In these scenario, the model underestimates sales at times when neither value of radio investment or newspaper investment are extremely small or high. i.e, when they both have moderate values and contribute significantly to Y.
The model overestimates sales at times when either newspaper or radio has extremely high value i.e, newspaper investment being high and radio investment being low or vice versa. (When one particular predictor value is high).
This is because, Sometimes in reality, the resultant target would have a high value when all the predictors contribute to the prediction instead of just one predictor contributing more and the others contributing less.
This phenomenon where the original sales is high when both the radio and newspaper contribute moderately well , rather than when only newspaper or radio investment is being an extreme contributor , is called synergy or interaction effort.
Example for better relatability : Extremely high number of workers and very low number of resources will not lead to much progress.
Just like that , A lot of resources and very small number of workers will also not lead to much progress. we need both workers and resources at a good trade-off. This is also an example of synergy effect and a lot of real life situations is also like this, rather than simply being a linear relationship.
What is Missing with our Original Model ?
Coming back to our Advertisement data-set, what is our model missing ?
Y = beta0 + beta1*radio + beta2* newspaper
In this equation , The important thing that the model is missing is to account for the change in impact of a predictor variable on the prediction when the value of another predictor variable changes.
To put it more simply, when the investment on radio changes, the impact of newspaper investment on prediction Y also changes. but, our simple linear model fails to account for this change in impact of newspaper investment on Y based on the change in amount invested in radio.
Solution For Synergy Effect
This new model might do a better job on predicting the sales
Y = beta0 + beta1*radio + beta2*newspaper + beta3*(newspaper*radio)
So, how does this new model change the efficiency of the model ?
With some basic math, the above equation can be modified to show that
Y = beta0 + (beta1+beta3*newspaper)*radio + beta2*newspaper
When , The investment on newspaper changes, it not only changes the sales directly. but, it also changes the impact of radio investment on sales.
Instead of taking radio common in the above equation, if I took newspaper common, The vice versa could have also been shown easily.
Identifying Synergy
Identifying synergy between 2 variables can be done using a 3 dimensional plot between the 2 predictors and the prediction variable and also plotting the regression model plane on the same plot. if the model underestimates the original target values in areas where both predictors contribute significantly and overestimates the original target values where one of the predictor has extreme values, You can be certain that there is synergy between the variables.
In Those cases, using the above model might prove effective.
References
Introduction to Statistical Learning with Applications in R
