Measuring the efficiency of creativity in advertising has historically been challenging. It is hard to isolate the impact of creativity from other factors that impact performance, such as execution tactics or brand health. As the advertising landscape evolves, with brands using several creatives at once and with more and better data, it has become increasingly important to understand the impact of creativity…
Ekimetrics conducted a study providing a technical Marketing Mix Modelling (MMM) approach (object detection algorithms and multi-stage econometric modelling) that demonstrates an objective approach to creative measurement.
Here is an executive summary that highlights the important features and the most efficient ones regarding creative execution & marketing effectiveness by sector.
Scope & Methodology of the study
This study, which relies on Meta and focuses on Meta creatives, used AI to look at 3 billion impressions of nearly 23,000 individual creatives across three and a half years with 124 models encompassing multi-stage econometric modeling and Object/text Detection. We used a sample of five brands from the Insurance, Cosmetics, Hospitality and automotive sectors, and 13 related outcome KPIs. We noticed patterns in the features that make a creative driving a higher return on investment. We also spotted the irrelevant features. Indeed, the creative elements of a campaign are defined by features. Features are objects or sets of objects present in the marketing creative, all of which are identified by object detection algorithms. This includes common objects like people, as well as brand-specific objects, such as a logo or product. Each creative is then codified according to the set of objects, allowing for the creation of time series variables that can be modelled through MMM.
To get a deep dive in the methodology, read these articles on:
So, which features are important?
In this study, a feature is considered relevant if:
This table outlines the findings in this study regarding features that matter for each sector. Please note that within each sector one or more brands were modelled through numerous sub-models.
Features that matter for each brand
* Automotive results were achieved with further sub-modelling where for Person it does not test Person appearing Y/N but Person with a Face and without a Smile appearing vs not; for Person & Product it tests person without a Face & Product vs not.
4 features that significantly drove ROI uplifts across brands: Brand Cue, Logo, Product and Person
Brand Cue was significant at driving higher ROI in Hospitality and Insurance. It could, however, not be tested for Cosmetics and Automotive because of low impression levels (3% and 0% respectively).
Logo was significant in Hospitality and Cosmetics. For Automotive, 74% of creative impressions featured a Logo (either on the car, above text, or in the corner of the creative), making it difficult to separate the effect of Logo from other features appearing in the creatives. Therefore Logo did not achieve the expected significance. For Insurance, the result that a Brand Cue was relevant but not a Logo is interesting: either the audience does not associate the brand cue with the brand, or it is seen as a symbol of innovation with more appeal. Text was only significant for Cosmetics, despite it having good impression levels for all the other brands’ models. This could be due to the fact that Text was associated with the majority of the impressions (between 62-83%) for the other brands, making it difficult to truly determine the impact of text. It is worth noting that Text in Cosmetics could also be interpreted as a Product, as text would often be on the label (e.g. the scent description).
Person results (Person, Person & Face, Person & Face & Smile, Person & Product, Person & Face & Product) must be read with caution as they are not mutually exclusive. A person smiling was only relevant for the Insurance brand but not in the other brands. Other brands are more Product focused, with more creative impressions featuring products (against Insurance, with only 6% of creative impressions featuring products). It would appear, pending further validation, that when a brand sells ‘intangible products’ (e.g. insurance products) then smiles and people become more relevant. For Automotive, Person & Face and Person & Product were the only relevant features. For this brand, 83% of the creative impressions featured a Product and only 56% a Person. Creatives labelled as Product in this brand showed either a whole car, or very close-up images/videos of the inside of cars (with some displaying tech screens). These results would indicate (pending further validation) that adding a person to your product creative generates an uplift in ROI. This provides support for the research discussed in the ‘Object Selection’ section
The most efficient features are those which are relevant and have the highest ROI uplifts
Top performing features by brand
That is, the creatives that display a feature are much more efficient at boosting the ROI than all the other features or none of the features appearing. The table above outlines the top-ranking features in terms of ROI uplift for each brand.
The features with the largest uplift across brands (ranked 1) were Product, Text and Person with Face; Text was the most efficient in Cosmetics, Person with Face in Hospitality and Automotive and Product in Insurance. Due to the nature of Text in Cosmetics, this specific feature should be read with caution, however. When looking at the pattern of text along with the other features in Cosmetics, it is found that Text was always accompanied by a Product. For that reason, we cannot say whether Text in isolation drives higher ROIs, but Text with Product was seen to have a positive uplift. A sub-model containing just Product & Text features could be run in order to answer this question.
Excluding Text, the most efficient features were thus Product and Person with Face. These two features were also shown to be important in combination; the combination of Person & Product ranked second in both Cosmetics (with Face) and Automotive (without Face). Brand Cue was also a significant feature in the analysis, ranking sixth and seventh for Insurance and Hospitality, respectively. Similarly, Logo ranked sixth for Hospitality and Cosmetics. When combined, these two features were found to be even more efficient for Insurance and Hospitality, as Logo & Brand Cue was ranked second for Insurance and fifth for Hospitality. Logo Size was tested in a similar sub-model to just the Logo feature, but the Logo impressions were split out according to size: Small, Medium or Large (determined by the % of the creative which detected the logo occupied). The partner variable was the same as in the Logo sub-model: No Logo. No significant results were found for Large Logo due to the low volume of impressions associated with this feature (<3% for all brands).
For Hospitality, Medium and Small Logo were ranked second and fourth, respectively, while for Cosmetics, Medium and Small Logo were ranked fourth and fifth respectively. Both results indicate that having a bigger Logo is more efficient than having a Small Logo, and in general, having a Logo is better than having no Logo at all.
For reference only!
Many of the results could not be used because they did not meet the minimum impressions threshold (5%), or the uplift was <1 (meaning that a feature appearing was less efficient than not appearing). This was especially relevant for the Logo, Brand Cue & Text category. An interpretation of a <1 uplift is that splitting the creative impressions by having a Logo (Yes/No) did improve model performance and achieved significant coefficients to its splits, but the Logo split had a lower ROI than the not having that split. This does not mean that having a Logo in your creatives on Meta will decrease your ROI if you are an insurer, but rather that other features are more efficient than a Logo.
Testing the appearance of a feature vs its partner (e.g. Logo appearing vs not appearing) means that the partnering feature contains all other features. In the Insurance example, in the Logo sub-model, we are testing the appearance of a Logo vs ‘everything else’ (could be a Product, a Landscape, a Brand Cue, etc.). Does this mean that all other features are more efficient at driving sales than a Logo? No, there could be one single feature that is much more relevant than the rest and making the Logo partnering split more efficient.
In general, these results should be interpreted with caution and should always be considered in context of the brand and the creatives included in the study. For example, knowing that the creatives in the Cosmetics brand often included products with labels on them helps in understanding the results for this brand. Whilst most sub-models showed a positive uplift of the feature appearing, for Text, most sub-models results indicate that adding text to a creative would lead to a lower ROI.
At Ekimetrics, we help companies get more from their data and we implement pre-packaged AI solutions, so they can combine high impact with long-term business purpose. Our 400 data experts and industry specialists work together in integrated squads. We value specific industry and domain expertise and focus on useful, usable and used solutions.
Contact Ekimetrics’ data experts!
Thought Leadership
Thought Leadership
