Visualizing machine learning models and currency forecasts
•
1 j'aime•655 vues
This set of slides shared at PyCon 2017 Delhi talk about the need for visualizing black box models and the ways in which we can improve their understanding through interactions.
Recommandé
Contenu connexe
Similaire à Visualizing machine learning models and currency forecasts
Similaire à Visualizing machine learning models and currency forecasts (20)
Plus de Gramener
Plus de Gramener (20)
Dernier
Dernier (17)
Visualizing machine learning models and currency forecasts
- 1. 1 VISUALIZING MACHINE LEARNING PYCON 2017, DELHI
- 2. INCREASINGLY, OUR BEST MODELS ARE BLACK-BOX
- 3. 3 TELECOM CHURN “Churn of customers is a particularly severe problem in the telecom industry. The challenge is to identify the propensity of churn up to a month in advance, even before a customer moves out, so that proactive interventions can begin”
- 4. 4 OK WASTED Marketing cost Rs 40 MISSED Acquisition cost Rs 80 OK No churn Churn NochurnChurn Prediction Actual 8.3% 0.0% MISSED WASTED 6.61 COST PER CUST. 0.0% IMPROVEMENT Base MODELS
- 5. 5 Outgoing call 0 0 - 4 15+5-14 1 RECHARGE AMT > $20 01 YN > 1 RECHARGE 0 N Y 3.2% 3.6% MISSED WASTED 4.0 COST PER CUST. 39% IMPROVEMENT Decision Tree MODELS
- 6. 60.6% 2.5% MISSED WASTED 2.21 COST PER CUST. 66% IMPROVEMENT SVM MODELS OK WASTED Marketing cost $1.8 MISSED Acquisition cost $4.1 OK No churn ChurnNochurnChurn PredictionActual
- 7. PRICE FORECASTING FOR AN ASIAN AGRICULTURAL ENTERPRISE Problem Approach Outcome A Gramener Advanced Analytics Case Study A leading agricultural enterprise wanted price forecasts for their products in order to plan inventory release to optimise revenue. Incorrect timing was leading either to loss of revenue or unsold inventory. Gramener applied a suite of price forecasting models based on internal and external factors. The models were evaluated on multiple test datasets to select one that minimised median absolute deviation. The model was able to forecast the price to an accuracy of 88%. Within the first quarter of deploying the model, the revenue uplift attributable directly to pricing was +3.2%.
- 8. 8 A COMPARISON OF PRICE FORECAST ACCURACY OF PURE MODELS Product Moving Average Auto- regression Exponential Smoothing ARIMA Exponential Smoothing Over State Space Hybrid Model Neural Network Multi-Linear Regression Product 1 65.13 54.13 65.98 66.16 71.67 73.24 78.96 70.46 Product 2 66.89 56.66 66.74 68.12 74.41 74.65 89.15 73.87 Product 3 37.53 9.84 44.55 42.28 50.49 46.86 61.35 53.03 Product 4 37.16 4.92 50.22 43.50 52.19 53.40 68.63 53.15 Product 5 68.83 71.24 68.38 68.12 75.58 71.47 90.80 72.69 Product 6 69.41 69.60 69.24 70.16 77.55 75.75 80.41 75.09 Product 7 69.27 64.76 68.61 69.21 73.39 74.06 82.10 75.20 Product 8 64.54 52.50 63.93 64.41 68.31 70.82 79.70 70.78 Product 9 57.97 52.64 57.40 58.53 63.90 63.15 78.80 63.04 Product 10 53.61 55.90 54.54 56.47 59.78 58.63 90.28 61.96 Product 11 52.02 26.49 54.92 53.65 60.80 63.89 78.40 52.23 Product 12 45.83 28.50 53.59 49.43 56.09 53.63 85.34 48.33 Product 13 41.30 28.98 40.51 38.88 50.84 47.57 63.76 50.55 Product 14 41.14 17.41 41.51 38.05 45.95 48.69 71.55 44.10 Product 15 86.40 84.00 86.58 87.29 88.80 90.78 99.91 88.04 Product 16 85.76 83.83 85.66 85.59 85.30 88.43 91.76 78.59
- 9. WE NEED A WAY OF UNDERSTANDING BLACK-BOX MODELS
- 10. LET’S EXPLORE ONE BLACK-BOX MODEL K-MEANS CLUSTERING
- 11. WE UNDERSTAND THROUGH VISUAL SUMMARIES
- 12. 12 SEGMENTING INDIA’S DISTRICTS BASED ON BEHAVIOUR Previously, the client was treating contiguous regions as a homogenous entity, from a channel content perspective. To deliver targeted content, we divided India into 6 clusters based on their demographic behaviour. Specifically, three composite indices were created based on the economic development lifecycle: • Education (literacy, higher education) that leads to... • Skilled jobs (in mfg or services) that leads to... • Purchasing power (higher income, asset ownership) Districts were divided (at the average cut-off) by: Offering targeted content to these clusters will reach a more homogenous demographic population. Skilled Poorer Richer Unskilled Skilled Uneducated Educated Uneducated Educated Unskilled Purchasing power Skilled jobs Education Poor Breakout Aspirant Owner Business Rich Poor Rural, uneducated agri workers. Young population with low income and asset ownership. Mostly in Bihar, Jharkhand, UP, MP. Breakout Rural, educated agri workers poised for skilled labour. Higher asset ownership. Parts of UP, Bihar, MP. Aspirant Regions with skilled labour pools but low purchasing power. Cusp of economic development. Mostly WB, Odisha, parts of UP Owner Regions with unskilled labour but high economic prosperity (landlords, etc.) Mostly AP, TN, parts of Karnataka, Gujarat Business Lower education but working in skilled jobs, and prosperous. Typical of business communities. Parts of Gujarat, TN, Urban UP, Punjab, etc Rich Urban educated population working in skilled jobs. All metros, large cities, parts of Kerala, TN The 6 clusters are
- 13. 13 How to classify clients by behaviour Using customers’ ad spend patterns, categories of purchase, periodicity, price points and impact, Gramener accurately classified clients to 1. Offer personalised deals 2. Create new products Big buyers across categories at low price points P&G Cadbury Reckitt HUL 1 Big buyers across categories with better price & viewership Godrej L’Oreal ITC GSK J&J Amazon Coke 2 Mid-buyers across categories with avg price & viewership 4 Heinz Apple Future Group LIC Ford Amul Large clients Medium clients Small clients Tiny clients Size legend Each box contains a cluster of advertisers with similar behaviour FMCG Auto Telecom E-commerce Electronics Retail BFSI Infrequent Hindi Movie ads in regular slots at high price 5 Getit TVS Quickr Lenovo HPAircel Axis MRF Microsoft ICICI Ceat Motorola Infrequent Hindi GEC advts with high TVR/ very low price 6 Saavn Voltas PNB Birla Sunlife Jivraj Tea Pitambari Summercool Home Appliances Frequent regional channel ads with low viewership 7 Pepperfry Shoppers Stop Bank of Maharashtra Raja Biscuits Cookme Spices Pran Foods Dipros Metro Dairy Koel Fashions Meghbela Big buyers across categories with low regional advertising 3 Nestle Maruti Airtel OLX Samsung Dabur Occasional Hindi GEC advts at moderate price points United Biscuits 8 Expedia BigBasket Sulekha Union Bank Yes Bank Piaggio BMW Hitachi Occasional regional and Hindi GEC ads at high price 9 PayTM Franklin Templeton Duroflex Mother’s Recipe Anchor Electricals Advertiser Clustering Transform variables to minimize correlation Cluster customers to minimise overlap Profile clusters to interpret their characteristics
- 15. 15
- 16. 16 LET’S EXAMINE CURRENCY FORECASTS
- 17. 17 68% correlation between AUD & EUR Plot of 6 month daily AUD - EUR values Block of correlated currencies … clustered hierarchically
- 18. WHAT YOU SHOULD TAKE AWAY BLACK-BOX MODELS ARE INCREASINGLY ACCURATE BLACK-BOX MODELS NEED INTERPRETATION (EVEN MORE) BUILD VISUAL SUMMARIES TO EXPLAIN MODELS MOVE UP & DOWN THE LADDER OF ABSTRACTION TOOLS ARE LESS IMPORTANT THAN TECHNIQUE
- 19. 19 THIS IS GRAMENER’S STACK PythonR JavaScriptExcel Pandas Tornado d3 lodash Pivots VBA ggplot2 plyr Analyse Communicate External Internal We recruit across this stack (and there’s a skill gap in the market in each of these)
Notes de l'éditeur
- A decision tree is a visual representation of choices, consequences, probabilities and opportunities. They are visual representations of the average outcome. Applying the same fundamental to predict the churn handling we were able to calculate the cost per customer and improvements which were done. On an average, whether an outgoing call was made from the phone. In case of a viable answer, we were able to fix 3 buckets of 0-4 days, 5 to 14 and more than 15 days. If the call has not been made for more than 15 days, there will no recharge voucher applied and the customer may likely leave the network. In cases where the call has been done within 15 days and the simultaneous recharge has been done only once, what has been the recharge amount. If amount is greater than 50, the loop starts from beginning and we establish that the consumer is engaged and spending will not be huge. Earlier, the telecom operator for whom the design has been done was spending more. The decision tree helped them to save 62% of their costs with only 3.2% of cases on an overall basis.
- So, what we did was put a variant of this visual together. On the right, you have a series of currencies like the Australian dollar, the Euro, the British pound, etc; some commodities like silver and gold; and some stock indices like Sensex, FTSE, and S&P. The cells here have a number inside that indicates the pairwise correlation between a pair of securities. For example, the number 68 on the top left indicates a 68% correlation between the Australian dollar and the Euro. To the left of the Euro and just below the dollar (diagonally opposite to the 68), there’s a scatter plot that shows the daily prices of both these currencies. Each dot is one day’s data. The x-axis shows the Australian dollar value. The y-axis shows the Euro value. This helps identify what the pattern of movements of any two currencies is. From this, you can easily see visually that the Australian dollar and the Euro both tend to move together. Or, where there are strong correlations like the FTSE & S&P, the pattern is almost a straight line. In some cases there are negative correlations. For instance, if you take the Sensex against the Japanese Yen, the correlation is -79%. The cells are coloured based on their correlation values. Greens indicate strong positive correlation. Reds indicate strong negative correlation. These are also grouped hierarchically. On the left, we have a series of lines indicating clusters. The most similar securities are grouped together. So FTSE and S&P with a 98% correlation are very close. The ones that are less correlated are kept further away based on a tree-structure. This leads to clustering of securities. For example, there is a green block in the center which has SGD, JPY, XAU, CHF and CNY. All of these are fairly well correlated. When any one currency in this block goes up, all the others go up as well. When any one goes down, all others go down as well. Similarly, you have another block to its top left: S&P, FTSE, Sensex and to a certain extent, the Pakistani Rupee. These move together as a block as well. But when this block goes up, all the currencies in the other block go down, as indicated by the red negative correlations between these two blocks. This can be used very easily for decision making. For example, one client who was trading with Singapore and Japan looked at the strong correlation and decided to consolidate their holdings in Japanese Yen. They then moved up and down this column to find a good hedge. FTSE looked like a good hedge – it was the most negatively correlated with JPY at that time -- and they decided to place a third of their portfolio in FTSE. A sheet like this improves people’s understanding of relatively complex data, and results in significantly increased trade volumes.