Machine Learning: Friend or Foe?
•Télécharger en tant que PPTX, PDF•
1 j'aime•281 vues
From Click Consult's Benchmark Search Conference 2018, Hilton Manchester Deansgate, 5th September. Presented by Dave Karellen, Head of Paid Search, Click Consult
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (13)
Similaire à Machine Learning: Friend or Foe?
Similaire à Machine Learning: Friend or Foe? (20)
Plus de Click Consult (Part of Ceuta Group)
Plus de Click Consult (Part of Ceuta Group) (20)
Dernier
Dernier (20)
Machine Learning: Friend or Foe?
- 1. Machine Learning: Friend or Foe? Dave Karellen Head of Paid Search www.click.co.uk david.karellen@click.co.uk
- 2. Split Opinions www.click.co.uk Machine Learning and Automation polarise the industry Against For “Agencies who won’t adopt AI technologies will simply vanish from the market.”
- 3. Automation Is Nothing New www.click.co.uk Leaning too much towards either extreme can be equally dangerous
- 4. Machine Learning www.click.co.uk The key component is a feedback loop that lets the tool ‘learn’
- 5. Problem 1: Lack of Inputs
- 6. Ad Rotation Settings www.click.co.uk Pre September 2017 Post September 2017 This leaves a lot of unanswered questions
- 7. Unanswered Questions www.click.co.uk What level of significance should we use? What if conversions have different values? Can we set min and max caps of share of voice? How do we deal with sale ads? Why can’t we optimise by ‘engaged clicks’? I want to be alerted when I should replace an ad. What themes have worked best?
- 8. Often It’s Better To Build Your Own www.click.co.uk Using your own or 3rd party confidence testers you can fulfil your individual ad optimisation requirements.
- 9. Problem 2: Black Box Processes
- 10. No One Knows What Is Inside www.click.co.uk How can you decide which to pick?
- 11. Evolution of Bid Strategies www.click.co.uk We’ve come a long way, but do we have any more insight on how it works?
- 12. Broad Keywords– The Original Black Box www.click.co.uk Keyword: red dress Increasing bid | Decreasing relevancy red dresses maroon dress peplum dresses red clothes women’s clothes green skirts Increasing bid | Decreasing relevancy Keyword: samsung case samsung phone chargersamsung phone cases iphone case htc charger flip casesred samsung case case for s9 red night dress
- 13. “No Way To Control It… It’s Totally Automatic” www.click.co.uk But does it always have to be this way?
- 15. Problem 3: Levels The Playing Field Too Much
- 16. If Everyone Uses The Same Methods, Who Wins? The cartoon equivalent of two advertisers using Target Outranking Share
- 17. Lack of competition isn’t always a bad thing Sometimes everyone can win, but this is rarely true in advertising
- 18. Use Tools To Convert Search Term Reports To Search ‘Word’ Reports It’s easier to spot trends on aggregate data, and can be used to inform ad copy and other channels
- 19. Problem 4: Measurement Has Not Caught Up
- 20. www.click.co.uk Returning vs New Acquisition Are we measuring the correct difference in lifetime values between new acquisitions and returning customers?
- 21. Advocacy Products www.click.co.uk Certain products are more likely to turn people into brand advocates, so should we attach more value to these purchases?
- 23. So Near, Yet So Far www.click.co.uk There is still no such thing as a ‘holy grail’ of attribution Budget Revenue £1000 £3000 £3000 £8000 Additional £5000 Revenue Maximising Total Profit Channel Budget Last Click Channel Profit Attributed Profit Total Profit Channel A £6,415 £9,664 £9,463 £19,126 Channel B £3,163 £5,240 £6,034 £11,274 Channel C £9,422 £10,638 £6,195 £16,833 Totals £19,000 £25,542 £21,692 £47,234 SEO Email PASS DROP
- 24. Leveraging Custom Rules With DDA This can bridge the gap for areas Data Driven won’t measure on its own
- 25. Recap Problem 1: Very limited inputs available Solution: Build your own tools instead – not worth it Problem 2: Process is too ‘black box’ Solution: Test to make sure it matches your expected outputs. If possible make own transparent tools Problem 3: No competitive gain opportunity Solution: Automate menial tasks, but use human touch to make competitive gains and add creativity. Problem 4: Measurability lags behind Solution: Accept machine learning in general, but overlay your own measurement from external analysis. www.click.co.uk
- 26. Thank You Dave Karellen Head of Paid Search Email david.karellen@click.co.uk Website www.click.co.uk Any Questions?
Notes de l'éditeur
- Hello Manchester! I’m Dave Karellen, Head of Paid Search at Click Consult. Today, I’m going to be talking about machine learning within PPC, and presenting an unbiased view of the topic. As always, rather than just presenting a viewpoint, my main aim is to be able to give you insights that are actionable so that you can take them home to improve your accounts. Now, I say an ‘unbiased view’ as I very much want to give an honest appraisal of the problems with the approach the industry is currently taking to implementing machine learning. Machine learning can be one of the most powerful tools, but often it is misunderstood by people.
- Generally people are split into two very distinct camps when it comes to machine learning adoption. On one side you have the tin foil hat wearers who are -at best- concerned that machines will take all jobs away, and at worst think judgement day is around the corner. On the other side are the people who will lap up any new buzzword they hear, often confusing what the actual concept is. On their extreme, are people sporting fading tattoos of ‘I <3 Big Data’ and whose old calendars are all filled with ‘This is the Year of the Mobile’ from 2011 through to 2016. And these are the people who actually claimed to like the new AdWords interface when it first came out.
- This is nothing new and history is littered with examples of where automation has disrupted technology from Henry Ford to Steve Jobs. The fate of those that ardently oppose automation is well documented, but those that didn’t appreciate the need for a balanced approach tend to fail just as spectacularly. Recently, Elon Musk has had to concede – amongst other things in the news, that his approach for building the new Tesla Model 3s was too heavily reliant on machines, and admitted that ‘humans are underrated’. It seems too much or too little automation is equally detrimental.
- So what exactly is Machine Learning? Machine learning is different from standard automation processes as the tool improves its ability to complete tasks based on how well it fared in previous attempts. This allows the tool to effectively learn, by using feedback loops to inform it. Nothing more complicated than that at its heart, no magic or tricks. On its own machine learning represents a massive step forward for automation and improving performance. Machine learning has the potential to empower advertisers who are determined to take their accounts to the next level, by advancing tasks such as bid optimisation. To be very clear, I am 100% for Machine Learning when it is considered in this pure form of the idea.
- However, this isn’t the only way in which Machine Learning is currently presented to marketers, and I’d like to begin discussing the challenges faced to adoption, starting with considering the inputs we have available. Most examples of machine learning in AdWords are found in the automated bidding strategies and ad rotation settings.
- If we look first at the ad rotation settings available in AdWords. Up until September 2017, we had the following ad rotation settings available. I wouldn’t have said this was a great choice, particularly, but at least preferable to post September 2017 when we are left with the following. A binary choice! That’s all we have to pick from. Are we for machine learning or against? How limiting is this? The optimise setting will favour ads which are deemed by Google to perform better- generally in terms of click-through rate, but also conversions if using a bid strategy. But you see, the problem here is that there are lots of inputs that we would ideally want to use to control this, or questions we’d want to be answered.
- For example, what level of statistical significance is used? Is it 90%, 95%? What if we wanted to decide to optimise faster or instead more precisely? If we conjoin it with a bidding strategy, what if some conversions have a greater value than others, how do we compare micro conversions and macro conversions separately? If there are sale ads, what if we want to ensure they get a decent share of voice? Ultimately we don’t want ads to get stale, even if the best ad is found out of the original set of three, the whole point is to test new variations, but we aren’t told when to. Machine Learning could help us identify themes for a knowledge bank to help inform these future tests. But we don’t have these choices available to us. So until we have more inputs to match your business strategy then Google’s solution isn’t complete. You need to look at ways of integrating machine learning with your own tools and cannot completely depend on it as it stands.
- So instead, it’s best to build your own tools or use a third party. Our one, for example lets us set our own confidence level depending on how quickly we want to run the test for. Notifications alert us when a winner has been found, so we can replace the other ads. We are using metrics important to us, in this case profit to eligible impression rate, which is more powerful than CTR. And we have a knowledge bank to help us decide which themes to use in new ad copy. There’s more on the creation of these later.
- And it’s not just a lack of inputs which is the problem. The second biggest issue surrounds the process itself. Machine Learning in PPC has become synonymous with black box activity. Namely that exact information on how the algorithms at work is kept under lock and key. To an extent this makes sense, neither Google nor any third parties really want to disclose all of their intellectual property, it is very much the same case as with SEO where the exact nature of algorithms is kept secret. However, for PPC these are tools that are supposed to help us, but it is taken to a point where advertisers don’t know what to expect in terms of performance from the tools on offer.
- Anyone can come up with black box tools claiming all manner of superlative performance boost, but without knowing the processes behind these, how would you know which one to pick? Black boxes systems don’t allow us the control we need as advertisers. The only way for us to know which tool would work best for us is by testing them, which is definitely the best approach. But if we knew a bit more about the systems at work then the more sophisticated advertisers would be able to determine which tools would be of the biggest help in each scenario, and save time.
- One of the biggest examples of a ‘Google Black Box’ was the old school ‘Conversion Optimiser’, affectionately known as ‘Conversion Floptimiser’. It used all manner of black box magic to all but guarantee poor to mediocre performance, with no one really knowing exactly how it worked, or even if it did. AdWords’ automated bidding strategies are much more advanced and indeed varied than the dark days of ‘Conversion Floptimiser’. However, there still isn’t much insight into the inner-workings of them.
- To give another example of a ‘Google Black Box’, few advertisers would be prepared to invest in a pure broad strategy due to the inherent ambiguity in exactly how far-removed the search terms may be from the original keyword. *ad lib example* And you’ll be hard-pressed to find true PPC specialists that were thrilled about the close variants updates. It is no different with the new bidding strategies. But, for those who aren’t willing to commit that level of trust towards the strategies, it is still possible to find the compromise between man and machine without resorting to a black box system. It all comes back to control.
- Anyone familiar with the Pointer Sisters’ 1983 song ‘Automatic’? *ad lib over 40 joke*. Well one of the lyrics is ‘No way to control it, it’s totally automatic’. However, despite the Pointer Sisters’ clear authority on the subject, I don’t believe that control and automation need to be mutually exclusive.
- The best systems can be transparent and easily understandable, and I’d like to show you one we use ourselves for bid optimisation. *ad lib* Tools such as these represent the workaround to bridge the gap between automation and control, so that human element can be integrated. As with Machine Learning, feedback loops ensure these tools evolve and adapt. Anyone with a good Excel knowledge can build these tools and can take them further with Scripts, but the philosophy of retaining control and transparency must remain at the heart.
- The next issue facing the direction of machine learning is around competitor strategy differentiation. Which basically means, if everyone adopts similar bidding strategies then who is winning?
- Target outranking share is the most obvious example of when nobody wins if it’s implemented by two competing advertisers, which essentially is two advertisers each saying “I want my bids to be higher than his!” The only person winning here is Google with inflated CPCs. In this example, you can think of Google as the gun manufacturer. Machine learning doesn’t always require you to be looking to one-up the competition though.
- Look into the great work by TensorFlow using Machine Learning to aid cancer detection. In this case everyone wins, and there’s no problem with that. However, this is advertising, and is driven by competition. There’s always ways to differentiate yourself with creative ad copy, but this shouldn’t be the only avenue available. It’s fine to automate menial parts of account management, but not cornerstones where you can make real competitive gains. Advertisers want to add value, do we really want a scenario where you just have it dumbed it down to an AdWords Express style set-up. Who wins? Are we in danger of levelling the playing field too much? To make real gain over competitors you need to be able to uncover trends for yourself. Even if ML is already making use of these trends, by knowing them yourself, you may find they’re useful outside of just PPC.
- One of the more creative ways to get actionable trends from your business data is through analysing Search Term Reports for word trends rather than whole terms. The new AdWords interface shows you some of these top words on the overview page, but it isn’t downloadable for all words. Instead you need to use Excel tools to aggregate word performance for every word that has appeared in a list of search terms. This can help you spot trends which may not have been clear with diluted data. This can be used to inform what terms might be worthwhile to use not just as keywords, but in ads, landing pages, and indeed other marketing channels such as email. Again this connects back to the system being black box unless we build tools such as these. But this more than anything represents a clear way to gain a competitive advantage by looking at the right trends that aren’t necessarily being examined by other advertisers relying on only completely automated solutions, so that you can impact the wider business.
- The final point I’d like to cover on where we have to be careful with Machine Learning is in situations where measurement hasn’t quite caught up. Even with the ability to import offline conversions and call tracking to aid with bid strategies, it still can’t paint the full picture.
- Let’s consider some examples. If someone new to your business converts at a £100 sale compared with a returning purchaser you also converts with a £100 sale, with both acquired at the same cost- which one actually represents a greater lifetime value to the business? Acquiring a completely new user who may come back to convert again would generally have a higher lifetime value than a previous converter who is already loyal. The exact difference in lifetime value is something you’d have to measure and decide yourself, and isn’t something that would generally be automatically factored into an automated bidding strategy. It doesn’t help that once you’ve figured this out, it isn’t possible to integrate audience bid modifiers with bidding strategies in a useful way. Again this goes back to the earlier points with lack of control in black box systems not allowing us to add our own business value. While we can manually measure lifetime value, there are even more difficult customer value differences to consider.
- Now, every retailer will have certain products that their customers will absolutely love- more so than the rest of their product base. These products are more likely to turn customers into brand advocates who will not just remain loyal, but will tell all their friends about your brand, driving more business. Now this is very difficult to measure, you can get an idea of a product’s potential to turn customers into advocates, let’s call them ‘advocacy products’, by looking at which products get the best reviews, however it is very difficult to measure their overall uplift. It’s not just cross-device attribution; this is cross-user attribution. But it is both a real thing, and definitely not negligible. It’s a similar concept to loss leaders being used to attract more customers. As it stands, the best way to integrate this into your strategy is to test high positive review products in higher positions, and look at how this impacts your bottom line, but this isn’t something that is considered by machine learning algorithms- yet.
- With attribution, obviously the biggest step towards machine learning has been Data Driven Attribution, which uses Machine Learning to determine how best to attribute value in multi-step conversions. This has been one of the most divisive developments in AdWords, some incredibly in favour, and others totally against. Now to be clear, I fall very much into the ‘incredibly for it’ category. However, I would definitely stop short of saying it is the ‘holy grail’ of attribution, and the main problem is that we’re still a million miles away in measurement ability, as illustrated by the previous examples.
- Two years ago, I gave a speech about attribution, detailing how the best approach is to find as many customer insights for your business as possible and start building up your own attribution rules based on this- testing each against bottom line performance. At the time, I thought it would be a lot longer than it has been before we got to a stage of having a system as strong as Data Driven Attribution. Anyone who isn’t using this when it’s available, perhaps due to not having buy-in from upper management, must understand that last click’s reign is ending. However, even more important, is that anyone who believes that this on its own is the ‘holy grail’ of attribution is equally mistaken. Despite data driven coming faster than I for one expected, this doesn’t mean that measurement has anywhere near to 100% caught up.
- As such, we still need to consider overlaying custom rules, and this is possible within GA360. The Data Driven Attribution model can be used as a baseline, and you can set up your own custom rules on top of this, for example based on whether someone is likely to become an advocate or if it’s a new acquisition, like in the previous examples. Using your own findings in conjunction with data-driven attribution helps you regain some of the control, and lets you make a competitive edge in your vertical. As always, make sure you test this against the bottom line. Just never think that data driven on its own will give you the best results; it’s a good baseline but until measurement ability catches up, the human touch is still needed here.
- So to summarise, machine learning represents one of the biggest opportunities for the future of PPC if it is used correctly. We should all embrace it into our strategies, but watch out for the four main problems when machine learning is bundled in with these scenarios: Only limited inputs are available -best to shun in these cases and build your own tools System is too black box -test thoroughly to make sure it matches your expected outputs, again though if possible look to make your own transparent ML tools No chance for competitive gains -be creative to look for areas that no one has thought of before. Automate menial tasks in all cases, but never delegate areas where you can make competitive gains and add creativity. And when measurability lags behind- in this case, accept machine learning as helpful but realise its limitations and overlay your own measurement from external analysis.
- Thanks everyone. I’d be happy to take any questions, or if you’d like to speak to me more about machine learning, I’ll be in the afternoon panel session. Thanks!