Implementation challenges in Big Data - Dr. Nilesh Karnik

•

0 j'aime•1,236 vues

In todays competitive environment companies are faced with different types of challenges. Implementation of Big Data is one of them. Dr. Nilesh Karnik takes us through some of them.

Technologie Business

Aureus Claims Solution

Implementation
Challenges in Big Data
Footer Option 2
Analytics
• Dr. Nilesh N. Karnik

Copyright 2013 RESTRICTED CIRCULATION

What we will discuss

The Challenge of BIG Data

ADVANCED Analytics
SOLUTIONS in the Pipeline

Copyright 2013 RESTRICTED CIRCULATION

2

Big Data : Distributed Processing

Aureus Claims Solution

!
Footer Option 2

OLD IDEA
Copyright 2013 RESTRICTED CIRCULATION

NEW IDEA
3

EXAMPLE 1: Task of storing books on a shelf

Aureus Claims Solution

Footer Option 2
Image source Flickr. Image copyright belongs with original artist.

Simple, right?
Copyright 2013 RESTRICTED CIRCULATION

5

EXAMPLE 1: Task of storing books on a shelf

Aureus Claims Solution

And now?

Footer Option 2

Image source Flickr. Image copyright belongs with original artist.
Copyright 2013 RESTRICTED CIRCULATION

6

Aureus Claims Solution

Footer Option 2

Image source Flickr. Image copyright belongs with original artist.
Copyright 2013 RESTRICTED CIRCULATION

7

EXAMPLE 2 : Summarizing a Report

Aureus Claims Solution

SUMMER PROJECT
REPORT

Footer Option 2

Simple, right?
Copyright 2013 RESTRICTED CIRCULATION

8

EXAMPLE 2 : Summarizing a Report

Aureus Claims Solution

Footer Option 2

And now?
Copyright 2013 RESTRICTED CIRCULATION

9

EXAMPLE 3 : Baking a Cake

Simple, right?

Aureus Claims Solution

And now?

Footer Option 2

Image source PINTEREST. Image copyright belongs with original artist.
Copyright 2013 RESTRICTED CIRCULATION

10

Advanced Analytics
•

Well developed tool set for “small data” environment

•

Aureus Claims Solution

Challenges in Big Data environment

Footer Option 2

Copyright 2013 RESTRICTED CIRCULATION

11

Advanced Analytics: MapReduce Difficulties

Aureus Claims Solution

Footer Option 2
ITERATIVE
Image source Flickr. Image copyright belongs with original artist.
Copyright 2013 RESTRICTED CIRCULATION

12

Advanced Analytics: MapReduce Difficulties

Aureus Claims Solution

Footer Option 2
INCREMENTAL PROCESSING REQUIRES RESTART
Image source Flickr. Image copyright belongs with original artist.
Copyright 2013 RESTRICTED CIRCULATION

13

Advanced Analytics: MapReduce Difficulties

Aureus Claims Solution

Footer Option 2

BATCH LEARNING SCANS ALL DATA IN ONE GO
Copyright 2013 RESTRICTED CIRCULATION

14

Some Solutions Data Scientists are working on

Aureus Claims Solution

New frameworks
• E.g., HaLoop*, PrIter# (Extensions of Hadoop)
• Percolator$ (Proprietary Google framework)

Footer Option 2

* Y. Bu, B. Howe, M. Balazinska, and M. Ernst, “HaLoop: Efficient iterative data processing on large clusters”, VLDB, 2010.
# Y. Zhang, Q. Gao, L. Gao and C. Wang, “PrIter: A distributed framework for prioritized iterative computations”, SoCC, 2011.
$ D. Peng and F. Dabek, “Large-scale incremental processing using distributed transactions and notifications”, OSDI, 2010

Copyright 2013 RESTRICTED CIRCULATION

15

Some Solutions Data Scientists are working on

Aureus Claims Solution

Smarter algorithms / Different
implementations
• Random forest
• Parallelized Stochastic Gradient
Descent

Footer Option 2

Copyright 2013 RESTRICTED CIRCULATION

16

@nilesh_karnik
Nilesh@aureusanalytics.com
Copyright 2013 RESTRICTED CIRCULATION

17

Aureus Claims Solution

ThankOption 2
Footer You!
SINGAPORE

INDIA

Aureus Analytics Pte. Ltd.
17, Phillip Street,
#05-01, Grand Building
Singapore (048695)

Aureus Analytics Pvt. Ltd.
706, Powai Plaza
Hiranandani Gardens, Powai
Mumbai – 400076

info@aureusanalytics.com
Copyright 2013 RESTRICTED CIRCULATION

www.aureusanalytics.com

Contenu connexe

En vedette

Point of Decision Analytics in Insurance

Aureus Analytics

Work Samples - ASB Digital Savvy Workforce (2013)

Arturo Pelayo

Building Healthier Communities: TEDMED 2016

Luminary Labs

Life@SlideShare

Amit Ranjan

TIME: 2015 Person of the Year

guimera

Net Promoter Score Pitfalls to Avoid

Aureus Analytics

Digital Portfolios

Silvia Rosenthal Tolisano

Culture is something we take pride in at LinkedIn. As the collective personality of our organization, it sets us apart, defines who we are and shapes what we aspire to be. Hundreds of companies have defined their unique cultures on SlideShare as part of the Culture Code campaign. We thought it was important for LinkedIn to join in this effort; we want everyone, including our current and our future employees, to know exactly what it’s like to work here.

LinkedIn’s Culture of Transformation

Pat Wadors

En vedette (8)

Point of Decision Analytics in Insurance

Work Samples - ASB Digital Savvy Workforce (2013)

Building Healthier Communities: TEDMED 2016

Life@SlideShare

TIME: 2015 Person of the Year

Net Promoter Score Pitfalls to Avoid

Digital Portfolios

LinkedIn’s Culture of Transformation

Plus de Aureus Analytics

Machine learning and artificial intelligence have great potential in helping insurance companies improve their internal processes such as underwriting, cross selling, fraud and claims prevention which directly have an impact on the consumer experience. In this presentation Dr. Karnik shares his experience in building models for prediction of early claims and potentially fraudulent claims. These are based on working with insurance carriers of all sizes.

Using Machine Learning to Find a needle in a haystack Aureus Analytics

Aureus Analytics

Big Data and Analytics Trends 2018

Aureus Analytics

Improving Marketing Campaigns

Aureus Analytics

Improvement in Persistency Score for Large Insurer

Aureus Analytics

The Art Of Net Promoter Score

Aureus Analytics

Big data-analytics-trends-2016

Aureus Analytics

Statistical model infographic.compressed (2)

Aureus Analytics

Infographic: Big Data and Analytics trends for 2015

Aureus Analytics

Infographic: Steps to Building a Statistical Model

Aureus Analytics

Ten Commandments of Statistical Modelling - Infographic

Aureus Analytics

67 years of Steering Innovation

Aureus Analytics

Demystifying A Data Scientist

Aureus Analytics

FIFA World Cup 2014 in Numbers

Aureus Analytics

Big Data and Analytics Trends in 2014

Aureus Analytics

Plus de Aureus Analytics (14)

Using Machine Learning to Find a needle in a haystack Aureus Analytics

Big Data and Analytics Trends 2018

Improving Marketing Campaigns

Improvement in Persistency Score for Large Insurer

The Art Of Net Promoter Score

Big data-analytics-trends-2016

Statistical model infographic.compressed (2)

Infographic: Big Data and Analytics trends for 2015

Infographic: Steps to Building a Statistical Model

Ten Commandments of Statistical Modelling - Infographic

67 years of Steering Innovation

Demystifying A Data Scientist

FIFA World Cup 2014 in Numbers

Big Data and Analytics Trends in 2014

Dernier

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Top 10 Most Downloaded Games on Play Store in 2024

SynarionITSolutions

A Principled Technologies deployment guide Conclusion Deploying VMware Cloud Foundation 5.1 on next gen Dell PowerEdge servers brings together critical virtualization capabilities and high-performing hardware infrastructure. Relying on our hands-on experience, this deployment guide offers a comprehensive roadmap that can guide your organization through the seamless integration of advanced VMware cloud solutions with the performance and reliability of Dell PowerEdge servers. In addition to the deployment efficiency, the Cloud Foundation 5.1 and PowerEdge solution delivered strong performance while running a MySQL database workload. By leveraging VMware Cloud Foundation 5.1 and PowerEdge servers, you could help your organization embrace cloud computing with confidence, potentially unlocking a new level of agility, scalability, and efficiency in your data center operations.

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Principled Technologies

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Dernier (20)

Artificial Intelligence Chap.5 : Uncertainty

GenAI Risks & Security Meetup 01052024.pdf

Top 10 Most Downloaded Games on Play Store in 2024

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Strategies for Landing an Oracle DBA Job as a Fresher

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Automating Google Workspace (GWS) & more with Apps Script

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Axa Assurance Maroc - Insurer Innovation Award 2024

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

The 7 Things I Know About Cyber Security After 25 Years | April 2024

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Artificial Intelligence: Facts and Myths

HTML Injection Attacks: Impact and Mitigation Strategies

A Domino Admins Adventures (Engage 2024)

Boost PC performance: How more available memory can improve productivity

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Implementation challenges in Big Data - Dr. Nilesh Karnik

4. EXAMPLE 1: Task of storing books on a shelf Aureus Claims Solution Footer Option 2 Image source Flickr. Image copyright belongs with original artist. Simple, right? Copyright 2013 RESTRICTED CIRCULATION 5

5. EXAMPLE 1: Task of storing books on a shelf Aureus Claims Solution And now? Footer Option 2 Image source Flickr. Image copyright belongs with original artist. Copyright 2013 RESTRICTED CIRCULATION 6

9. EXAMPLE 3 : Baking a Cake Simple, right? Aureus Claims Solution And now? Footer Option 2 Image source PINTEREST. Image copyright belongs with original artist. Copyright 2013 RESTRICTED CIRCULATION 10

11. Advanced Analytics: MapReduce Difficulties Aureus Claims Solution Footer Option 2 ITERATIVE Image source Flickr. Image copyright belongs with original artist. Copyright 2013 RESTRICTED CIRCULATION 12

12. Advanced Analytics: MapReduce Difficulties Aureus Claims Solution Footer Option 2 INCREMENTAL PROCESSING REQUIRES RESTART Image source Flickr. Image copyright belongs with original artist. Copyright 2013 RESTRICTED CIRCULATION 13

14. Some Solutions Data Scientists are working on Aureus Claims Solution New frameworks • E.g., HaLoop*, PrIter# (Extensions of Hadoop) • Percolator$ (Proprietary Google framework) Footer Option 2 * Y. Bu, B. Howe, M. Balazinska, and M. Ernst, “HaLoop: Efficient iterative data processing on large clusters”, VLDB, 2010. # Y. Zhang, Q. Gao, L. Gao and C. Wang, “PrIter: A distributed framework for prioritized iterative computations”, SoCC, 2011. $ D. Peng and F. Dabek, “Large-scale incremental processing using distributed transactions and notifications”, OSDI, 2010 Copyright 2013 RESTRICTED CIRCULATION 15

15. Some Solutions Data Scientists are working on Aureus Claims Solution Smarter algorithms / Different implementations • Random forest • Parallelized Stochastic Gradient Descent Footer Option 2 Copyright 2013 RESTRICTED CIRCULATION 16

17. Aureus Claims Solution ThankOption 2 Footer You! SINGAPORE INDIA Aureus Analytics Pte. Ltd. 17, Phillip Street, #05-01, Grand Building Singapore (048695) Aureus Analytics Pvt. Ltd. 706, Powai Plaza Hiranandani Gardens, Powai Mumbai – 400076 info@aureusanalytics.com Copyright 2013 RESTRICTED CIRCULATION www.aureusanalytics.com

Implementation challenges in Big Data - Dr. Nilesh Karnik

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (8)

Plus de Aureus Analytics

Plus de Aureus Analytics (14)

Dernier

Dernier (20)

Implementation challenges in Big Data - Dr. Nilesh Karnik