The overall scores shown on the webapp are an average of 6 individual model scores, 5 of which cover creative elements (resnet, audio, imgII, transcript, ocr) and 6th for meta-tags.
For base metrics like Branding, Enjoyment, Persuasion, etc, is it possible to use the scores from the 5 individual models covering creative elements, see which has the highest and lowest score, and use that as an indicator of which creative element had the biggest role to play in the overall score being high / low?
We'll need to exclude the 6th individual model for metatags for this idea.
[Description: ../../Signature_Master_Logos_100pc/Signature_Master_logos_100pc_300dpi_Signature_Master_logo_Kantar.png]
Kelvin Lok, Senior Director, Engagement Manager
Analytics Practice
M +1 (781) 353 1715 E kelvin.lok@kantar.com
From: Lok, Kelvin (MaPS BOS)
Sent: Thursday, August 27, 2020 7:58 PM
To: Vinjamuri, Abe (Maps BOS) ; DAS, RAHUL (MB) ; Panigrahi, Pinak (MaPS CHE) ; L.R, Divyaa (MB)
Cc: Olson, Giulianna (MB) ; Krithivasan, Dinesh (MaPS CHE) ; K, Kugan (MaPS CHE)
Subject: RE: Fanta ads tested in LinkAI and Link
Got it - thanks Abe for the plan for the new norms.
For Paul, I'll let him know tomorrow that ...
* As the features extracted and ML model stands right now, there's not much we can do to make Link AI results line up more to Link.
* As smart features are added in as inputs to the ML model, let's see if that improves the ability to detect more of these things that the machine is missing in the present.
* Fundamentally though, based on what the machine is picking up, it is finding Colorful Snacking to be stronger than Colorful People so that's good. And with more differentiating elements being picked up in the future, we could perhaps see the difference magnified (doubt it will be as much as 10 ONS points per Link, but at least more than 1).
* I'll offer up to discuss over a call if that'd be helpful.
[Description: ../../Signature_Master_Logos_100pc/Signature_Master_logos_100pc_300dpi_Signature_Master_logo_Kantar.png]
Kelvin Lok, Senior Director, Engagement Manager
Analytics Practice
M +1 (781) 353 1715 E kelvin.lok@kantar.com
From: Vinjamuri, Abe (Maps BOS) >
Sent: Thursday, August 27, 2020 5:54 PM
To: DAS, RAHUL (MB) >; Panigrahi, Pinak (MaPS CHE) >; L.R, Divyaa (MB) >; Lok, Kelvin (MaPS BOS) >
Cc: Olson, Giulianna (MB) >; Krithivasan, Dinesh (MaPS CHE) >; K, Kugan (MaPS CHE) >
Subject: Re: Fanta ads tested in LinkAI and Link
We'll move to the new norms when we update the model. Either way those don't seem to be making as much of a difference if I'm seeing these results right. Which makes sense given norms change minimally.
Reg the 2 Fanta ads, I don't think we will solve Paul's core question of the spread in scores for the 2 ads. In this case I think the snacking ad (my subjective view) is better. But is it so good to move the ons by 40 plus percentile? That I am not sure. One could argue about the brand being shown more often and being integrated into the story (some of which we will have in the future with the smart features) that I turn drives easier understanding but am not so sure those 2 things should lead to the kind of differences in the two tests.
Get Outlook for Android
________________________________
From: Lok, Kelvin (MaPS BOS) >
Sent: Thursday, August 27, 2020 10:31:40 AM
To: DAS, RAHUL (MB) >; Panigrahi, Pinak (MaPS CHE) >; L.R, Divyaa (MB) >; Vinjamuri, Abe (Maps BOS) >
Cc: Olson, Giulianna (MB) >; Krithivasan, Dinesh (MaPS CHE) >; K, Kugan (MaPS CHE) >
Subject: RE: Fanta ads tested in LinkAI and Link
Thanks Rahul.
@Vinjamuri, Abe, I think you'd be aware we have new norms available for ONS calculations since May, and may already have a plan for if/when we switch the webapp from the current norms to the new norms?
On the Fanta ads, great to see that Colorful Snack comes out above Colorful People on ONS once we remove the metatag differences. The piece on ad length driving differences in results that are hard to explain is a good learning for us, and something to consider in the 80K model training.
But I guess for Paul, we still haven't really addressed the fundamental question which is that Colorful People scored 98 in Link but 102 in Link AI (so machine was not "low enough"), and Colorful Snacking scored 108 in Link but 103 in Link AI (so machine was not "high enough"). Paul got back on the sample, and said it was the standard (past 3 month consumption of carbonated soft drinks, age 12-49) so no reason to doubt the Link sample. Is there anything we can learn from the Link vs. Link AI scores in these 2 examples? (that's what Paul's going to ask)
[cid:image001.png@01D67D3C.29D8C470]
Kelvin Lok, Senior Director, Engagement Manager
Analytics Practice
M +1 (781) 353 1715 E kelvin.lok@kantar.com
From: DAS, RAHUL (MB) >
Sent: Thursday, August 27, 2020 9:30 AM
To: Lok, Kelvin (MaPS BOS) >; Panigrahi, Pinak (MaPS CHE) >; L.R, Divyaa (MB) >
Cc: Vinjamuri, Abe (Maps BOS) >; Olson, Giulianna (MB) >; Krithivasan, Dinesh (MaPS CHE) >; K, Kugan (MaPS CHE) >
Subject: RE: Fanta ads tested in LinkAI and Link
+Kugan
Hi Kelvin,
The attached excel sheet contains three tabs: old norms (that is currently present in webapp), the new norms (we downloaded this around May and we received from Kugan) and difference between old and new norms. In the new norms, for almost all the countries in 35k model (except "India - Combined Hindi Speaking Market - Offline" country) the norms information was available. For a few countries in 80k the country norms were not updated in LINK dashboard for certain ONS components. Regarding updating these norms in the webapp, the process is straight forward, we just have to replace the norms file. But we would like you to go over the old and new norms and give us your suggestion.
Fanta ads: We changed the ad length as 46s for both the ads and also we changed voice over to continuously for both the ads (voiceover intially was continuously for colorful people and no voice over for colorful snacking before). So the metatag predictions will now be same and the only thing that varies is the AI features impact captured by other feature models. By doing so, we get a score of 103 for colorful snacking and 102 for colorful people. Also, now the branding mean score value for colorful snacking is higher than colorful people.
We cannot turn off the metatag prediction to 0 in our final blending step, since the final model is a result from GBM, it is not a simple linear combination. But by ensuring the metatags are same for both ads (and therefore, the metatag predictions are same for both) we could quantify the effect of AI features alone in this case. When we complete our 80k model training, as a part of some experimental work, we are also planning to run a model without/very limited use of metatag. So in such instance, we can use those models to understand the effect of AI features.
Please let me know in case of any questions.
Thanks,
Rahul
From: Lok, Kelvin (MaPS BOS) >
Sent: 27 August 2020 02:22 AM
To: DAS, RAHUL (MB) >; Panigrahi, Pinak (MaPS CHE) >; L.R, Divyaa (MB) >
Cc: Vinjamuri, Abe (Maps BOS) >; Olson, Giulianna (MB) >; Krithivasan, Dinesh (MaPS CHE) >
Subject: RE: Fanta ads tested in LinkAI and Link
Hi Rahul,
Thanks again for taking a look on your end, and sharing the thoughts + attachments.
I added some comments in blue and questions in orange to your 2 bullets below. I think that will help us in our response back to Paul, and also to consider for our own learning (since Paul's objective for sharing with us these challenging questions isn't to cast doubt on our results, but to give us food for thought and improve our processes / models / results).
[cid:image001.png@01D67D3C.29D8C470]
Kelvin Lok, Senior Director, Engagement Manager
Analytics Practice
M +1 (781) 353 1715 E kelvin.lok@kantar.com
From: DAS, RAHUL (MB) >
Sent: Wednesday, August 26, 2020 6:58 AM
To: Lok, Kelvin (MaPS BOS) >; Panigrahi, Pinak (MaPS CHE) >; L.R, Divyaa (MB) >
Cc: Vinjamuri, Abe (Maps BOS) >; Olson, Giulianna (MB) >; Krithivasan, Dinesh (MaPS CHE) >
Subject: RE: Fanta ads tested in LinkAI and Link
Hi Kelvin,
We did look at the ads and we have a few thoughts.
1. ONS Calculation: We are not sure how Paul has calculated ONS so we went with our current approach. In the attached file you can see two tabs, one tab contains the ONS based on current web app norms "ONS_Webapp", another tab contains the ONS based on new norms "ONS_new_norms". In both the calculation, we are seeing Colorful snacking ONS score slightly lower than Colorful people
* Let's not doubt our ONS calculation, as we know it's correct per Link. Paul knows this as well, but still does his manual rough calculation every now and then.
* For my info, since it's my first time learning that we have current webapp norms vs. new norms:
* How are they different?
* Where did we get the new norms from?
* When will the webapp reflect the new norms?
2. Reason why colorful people ad has slightly higher branding score: We know that in our final model, metatag predictions have a big role to play. In this scenario, all the metatag variables are same for both the ads except for ad length. Colorful people runs for 60 sec where as colorful snacking runs for 46sec. From our dataset, we have seen ads close to 40-50sec have lower branding mean scores compared to ads with 60-70sec ad length. So this could be a reason why the branding for colorful snacking is lower than colorful people. Also, we looked at the prediction from individual models (image, audio, ocr etc). In most cases the branding for colorful snacking was slightly higher than colorful people, since metatag predictions has a higher weightage in the final models, we see the trend getting reversed.
* This might be worth us considering how we do the metatag predictions and then weight it in the final model.
* It seems like a fundamental problem if we can have ...
* Individual models for creative elements (image, audio, OCR, etc.) say that the longer ad (Colourful People) is stronger than the shorter ad (Colourful Snacking) in branding
* Then a single difference in the ad length metatag is enough to 1) cause such a big difference in individual model for metatags and 2) the individual model for metatags is weighted high enough in the blending to reverse the overall finding on branding
* It doesn't pass intuitive sense to say that Colourful People is higher on branding than Colourful Snacking just because it's at a sweeter spot in ad length
* Could we try re-running the components and ONS in the following ways:
* By making metatag for ad length the same (so we can see ONS and components without influence from the slight difference in ad length)
* By reducing weight of metatag predictions to 0 in the final models (so we can see ONS and components from just creative elements alone)
Attached are excel file (containing detailed predictions from each model, ONS scores) and plots (relationship between ad length and branding/ONS). Please let us know your thoughts.
Thanks,
Rahul
Cc'ing Dinesh in the thread.
From: Lok, Kelvin (MaPS BOS) >
Sent: 26 August 2020 03:38 AM
To: Panigrahi, Pinak (MaPS CHE) >; L.R, Divyaa (MB) >; DAS, RAHUL (MB) >
Cc: Vinjamuri, Abe (Maps BOS) >; Olson, Giulianna (MB) >
Subject: FW: Fanta ads tested in LinkAI and Link
Hi Pinak, Divyaa, Rahul,
Paul has reached out to us with another set of TCCC ads for Fanta to look into. See email chain from today for the details - it's not too long (yet).
Quick context:
* Two Fanta ads were tested in Link, and Paul put them through the UAT webapp in mid-July
* Ad 1 = Colorful People = content ID 100734
* Ad 2 = Colorful Snacks = content ID 100735
* Results are a table in Paul's original email at the bottom
* Colorful People did much worse than Colorful Snacking - Paul hypothesizes the Colorful People did so bad because it "reflects a reality that this whole 'colorful people may or may not drink Fanta' is a terrible idea! It's so hard to make a mood connect to a brand, but especially when your V/O doesn't even try to do so."
* Link AI results are quite similar for the 2 ads. Paul's ONS calculation would put Colorful People at a slight disadvantage behind Colorful Snacking (People = 101 vs Snacking = 102), but the ONS straight out of the webapp are different to Paul's and would say Colorful people is stronger by a few decimals and round to 1pp difference (People = 103 vs Snacking = 102)
Attached is the excel export from the webapp with the content ID, metadata, and full results.
Could you take a look through the chain below, and see if you have any ideas for how we could dive in? Paul and I shared a few bullets back-and-forth for some starter thoughts, and Paul will also get some feedback on the sample to see if the difference in Link is widened by sampling bias.
Kind regards,
Kelvin
[cid:image001.png@01D67D3C.29D8C470]
Kelvin Lok, Senior Director, Engagement Manager
Analytics Practice
M +1 (781) 353 1715 E kelvin.lok@kantar.com
From: Lok, Kelvin (MaPS BOS)
Sent: Tuesday, August 25, 2020 5:46 PM
To: McClean, Paul (MBATL cs) >
Cc: Olson, Giulianna (MB) >; Vinjamuri, Abe (Maps BOS) >
Subject: RE: Fanta ads tested in LinkAI and Link
I think we should be able to pull the video from the back-end. I found the video in the UAT webapp, so I'll share the IDs to the Chennai team and ask them to take a closer look using our thoughts below for a starting point.
And thanks for getting some feedback on the samples to see if there's any influence - that'll be good learning for us too.
[cid:image001.png@01D67D3C.29D8C470]
Kelvin Lok, Senior Director, Engagement Manager
Analytics Practice
M +1 (781) 353 1715 E kelvin.lok@kantar.com
From: McClean, Paul (MBATL cs) >
Sent: Tuesday, August 25, 2020 5:36 PM
To: Lok, Kelvin (MaPS BOS) >
Cc: Olson, Giulianna (MB) >; Vinjamuri, Abe (Maps BOS) >
Subject: RE: Fanta ads tested in LinkAI and Link
The two ads I ran the model on are in the UAT source, can you watch them from there ? I can send them if not.
I did calculate the ONS by hand, only because I wanted to check if it was the same as the AI calculation - the Colorful People ad was calculated by AI as 103 and by me as 101, neither of which is a great score, but also not the same. I used a rough guide based on the pattern of relationship between mean scores and index scores in the US database, and I know that isn't going to necessarily be 100% accurate. But with the final Link score at 99, a 103 would be further away, and maybe high given the scores for all but Branding and Enjoyment (though these have the largest weight in the calculation, of course). I have no reason to still do a by-hand calculation, really, so I'll stop that !
I'll get some feedback on samples as you suggest - could be an influence, yes.
Hopefully we can learn to add to the model.
Paul
From: Lok, Kelvin (MaPS BOS) >
Sent: Tuesday, August 25, 2020 5:17 PM
To: McClean, Paul (MBATL cs) >; Vinjamuri, Abe (Maps BOS) >
Cc: Olson, Giulianna (MB) >
Subject: RE: Fanta ads tested in LinkAI and Link
Hi Paul,
Thanks for sharing the Link results and your take on these 2 Fanta ads.
Would you be able to share the video files for these 2 ads (YouTube link is also okay)? It'll help in exploring further on our end.
Also, is there any way you can dig a little into the sample composition and open ends from the two Link tests? It'll help assess if sampling is magnifying the difference in Link results between the two ads (much wider than what the machine is finding, based purely on video and audio features).
Some quick thoughts from my end:
* Looks like you had to manually calculate the ONS on your end. Is the one on the dashboard still not accurate per your calculations?
* The machine agrees with Link that the Colorful Snacking ad is stronger than the Colorful People ad, which is good.
* It's just the difference in the Link results is much wider than the difference in the Link AI results ... I wonder:
* If the difference in Link results are "too wide" because of sampling or any other factor (could be as you outlined; that the core concept of Colorful People was horribly received compared to Colorful Snacking and so it drags all the scores down for Colorful People)
* If the difference in Link AI results are "too narrow", because of limited differences in the features extracted between the two videos, if the non-creative inputs are reducing the difference in final results, etc.
Cc'ing Gigi as well, so that we can all work on this together with the Chennai team.
[cid:image001.png@01D67D3C.29D8C470]
Kelvin Lok, Senior Director, Engagement Manager
Analytics Practice
M +1 (781) 353 1715 E kelvin.lok@kantar.com
From: McClean, Paul (MBATL cs) >
Sent: Tuesday, August 25, 2020 3:09 PM
To: Vinjamuri, Abe (Maps BOS) >; Lok, Kelvin (MaPS BOS) >
Subject: Fanta ads tested in LinkAI and Link
HI gents,
I think it's always good to review how our ads tested with both AI and Link look in terms of results. Here are a couple of new Fanta ads. I ran them through the AI model and they came out 'OK'. But in Link the results are somewhat different.
The ads are in the tool.
For 'People' the AI tool definitely over-stated Branding - I do see the Fanta bottle a few times, but the ad is really not well associated. And it also over-stated Understanding, which in Link probably reflects a reality that this whole 'colorful people may or may not drink Fanta' is a terrible idea ! It's so hard to make a mood connect to a brand, but especially when your V/O doesn't even try to do so.
Anyway, wonder what you make of these differences - I obviously recognize that we won't get a good match every time, but I wonder if there's anything we can learn here ? I'm thinking of what we were discussing the other day about non-creative components and their influence on the 'starting point' of any metric, for example.
For 'Snacking' the AI tool tends to under-state most metrics, especially Relevance and Brand Difference. Actually I believe that the core idea of Fanta being something that goes with Snacking is meaningful and pretty straightforward, but the machine didn't pick that up enough. Again, I wonder if there's any learning we can take out here ?
Ad Name
Enjoyment
Branding
Understanding
Relevance
Appealing
Brand Difference
ONS
Enjoyment
Branding
Understanding
Relevance
Appealing
Brand Difference
LinkAI
USA - Fanta Colorful People
3.67
4.18
3.53
3.09
3.76
3.73
101
105
103
93
97
101
102
Link
USA - Fanta Colorful People
3.72
3.83
3.25
3.19
3.75
3.85
98
106
91
77
101
101
106
0.05
-0.35
-0.28
0.1
-0.01
0.12
-3
LinkAI
USA - Fanta Colorful Snacking
3.74
4.16
3.58
3.04
3.78
3.64
102
107
102
96
95
102
99
Link
USA - Fanta Colorful Snacking
3.89
4.28
3.53
3.40
3.98
4.04
108
111
106
93
109
109
112
0.15
0.12
-0.05
0.36
0.2
0.4
6
[KANTAR]
Paul McClean
Senior Vice President
Coca-Cola Global Account Director
Insights Division
M +1 404-808-9269
3630 Peachtree Rd. NE, Suite 1200
www.kantar.com
[Twitter] [Facebook] [Google+] [LinkedIn]
Building Brighter Futures with UNICEF
Kantar Disclaimer
Guest
