Rapid Data Exploration With Hadoop

•

16 j'aime•1,828 vues

LinkedIn is the premiere professional social network with over 60 million users and a new user joining every second. One of LinkedIn's strategic advantages is their unique data. While most organizations consider data as a service function, LinkedIn considers data a cornerstone of their product portfolio. To rapidly develop these products LinkedIn leverages a number of technologies including open source, 3rd party solutions, and some we've had to invent along the way. This LinkedIn talk at the NYC Hadoop Meetup held 3/18 at ContextWeb focused on best practices for quickly uncovering patterns, visualizing trends, and generating actionable insights from large datasets.

Technologie

Rapid Data Exploration
With Hadoop
Peter Skomoroch
Senior Data Scientist

@peteskomoroch

Outline
• Overview: LinkedIn Biz, Tech, & Analytics
• Rapid Data Exploration 101
- Spatial Analytics Pig Code
- Trend detection with Pig & Python
- R Streaming Example
• Deep Dive: Our Data Analysis Approach
• Building Data Products
• LinkedIn Data Insights

Connect the world’s professionals to make
them more productive and successful

LinkedIn at a glance
• Founded in 2003
• #17 site in the US (Alexa)
• 60+ million members
• First million members = 477 days
• Latest million = 9 days
• 500K+ company proﬁles
• 12+ million small business professionals
• In 2009 - 1billion people searches
• Average age: 41
• Household income $107,000
• 42% are “decision makers”

How International?
• More than 50% international
(members in over 200 countries & territories)
• 13+ million in Europe
• 4+ million in India
• 3+ million in UK
• #13 site in UK (Alexa)

How do we keep the lights on?
• Proﬁtable since 2007
• Valued at over $1B at the last funding round
• Subscriptions
• Ads
• Job Postings
• Enterprise Client

Hadoop on LinkedIn
1,400+ members list “Hadoop” on their proﬁle
What other skills do they have?
•HBase, Lucene, Solr, MapReduce, Nutch...
Where are they? Who do they work for?
• 36% in Bay Area • 11% Yahoo!
• 8% in India • 2% Apache Software Foundation
• 6% in NYC • 1% LinkedIn
• 4% in Seattle • 1% Google
• 4% in Los Angeles • 1% Facebook

$Voldemort Data Storage Compact, compressed, binary data (something like Avro) Type can be any combination of int, double, float, String, Map, List, etc. => Sequence Files Example member definition: { ‘member_id’: ‘int32’, ‘first_name': 'string', ’last_name': ’string’, ‘age’ : ‘int32’ … }$

Getting Data In
•From Databases (user data, news, jobs etc.)
• Need a way to get data reliably periodically
• Need tests to verify data
• Support for incremental replication
• Solution: Transmogrify Driver Program
• InputReader: JDBCReader, CSV Reader
• Output Writer: JDBCWriter, HDFS writers
• From web logs (page views, search, clicks etc)
• Weblogs ﬁles are rsynced and loaded up in HDFS
• Hadoop jobs for date cleaning and transformation.

Giving Back: Open Source
http://sna-projects.com/sna/

We Build Things With Data

Give smart people great tools,
enable them to solve problems

How does Hadoop
enable rapid data
exploration?

R Streaming Also Easy

*from http://www.stat.uiowa.edu/~luke/classes/295-hpc/

Business is recognizing the importance of analytics

We can also leverage...
• Connection Graph • Company Pages
• Recommendations • Talent Match
• Address Book Uploads • Web Referrals
• Search Logs • 1M+ Twitter Accounts
• Proﬁle Views & Activity • Wikipedia Data
• Job Postings • Mechanical Turk
• LinkedIn Groups • Census, BLS, & Data.gov
• LinkedIn Questions • Much more...

How do we think of Analytics?

Data Jujitsu

Lots of Medium can be
more powerful than Big

>

Data Scientist Lessons
• Follow the data, avoid assumptions
• Sanity check the extremes (0, inﬁnity)
• Don’t get mired in rare edge cases
• Data Jujitsu: solve easier auxiliary problems
• Build smaller consistent samples to test code
• Establish a baseline model quickly, iterate often
• Use the right tool for the job at hand
• Iterate quickly with high level languages

We’re Hiring!
http://sna-projects.com/sna/
pskomoro@linkedin.com
@peteskomoroch

Contenu connexe

Tendances

Getting started in data science (4:3)

Thinkful

Getting started in data science (4:3)

Thinkful

Getting started in Data Science (April 2017, Los Angeles)

Thinkful

Data Science Overview

Davide Mauri

Demystifying Data Science with an introduction to Machine Learning

Julian Bright

Big data. Small data. All data. You have access to an ever-expanding volume of data inside the walls of your business and out across the web. The potential in data is endless – from predicting election results to preventing the spread of epidemics. But how can you use it to your advantage to help move your business forward? Data is growing exponentially and it’s now possible to mine and unlock insights from data in new and unexpected ways. Empower your business to take advantage of this data by harnessing the rich capabilities of Microsoft SQL Server and the familiarity of Microsoft Office to help organize, analyze, and make sense of your data—no matter the size.

Data Culture Series - Keynote & Panel - Birmingham - 8th April 2015

Jonathan Woodward

Slide From DataEngConf 2015 event. LinkedIn is the professional profile of record for our 400M+ members globally, but many people don't realize the full potential of their LinkedIn profile – especially on mobile. Adding blogs, photos and other rich content to your profile on a small screen device can get tedious. That's why LinkedIn created Satori, a Hadoop tool that crawls the web and extracts data to discover members' professional content online. Satori uses machine learning techniques and leverages other open source tools like Nutch and Gobblin in order to help match members with relevant content in order to maximize their professional profile. In this talk, Nikolai Avteniev, Sr. Staff Engineer and Agile Software Developer at LinkedIn, will share his experience in building the product and discuss the challenges and opportunities encountered along the way.

Building Satori: Web Data Extraction On Hadoop

Nikolai Avteniev

What Is GDS and Neo4j’s GDS Library

Neo4j

David golynskiy resume it5

David Golynskiy

Introduction to Graph databases and Neo4j (by Stefan Armbruster)

barcelonajug

Semantically Enabled Personal Information Management with Cluug.com

Bernhard Schandl

Graph processing and graph databases have been with us for a while. However, since their physical implementations are the same for every database in production (Node connected to node, or triplets), there's a perception that data modeling (and data modelers) have no role on projects where graph databases are used. This month we'll talk about where graph databases are a best fit in a modern data architecture and where data models add value.

Graph Databases - Where Do We Do the Modeling Part?

DATAVERSITY

Data Science: Harnessing Open Data for High Impact Solutions

Mohd Izhar Firdaus Ismail

AI in the Intelligent Workplace

Sharon O'Dea

Kurukshetra - Big Data

shankar_radhakrishnan

In recent years, the Internet industry has shifted more and more towards digital content distribution through online services. This presentation provides an overview of the overall system design and architecture of LinkSure News and Video Recommendations, the challenges encountered in practice, and the lessons learned from the production deployment of these systems at LinkSure. Specifically, we will highlight how news selection and personalisation of recommendations are formulated and addressed at LinkSure. By presenting our experiences in applying techniques at the intersection of recommender systems, information retrieval, machine learning, and statistical modelling in a large-scale industrial setting and highlighting the open problems, we hope to stimulate further research and collaborations.

Personalized News and Video Recomendation System at LinkSure

Leanne Hwee

Paving The Way To Data Driven

Mohd Izhar Firdaus Ismail

Smartlogic, Semaphore and Semantically Enhanced Search – For “Discovery”

VOGIN-academie

Big Data Content Organization, Discovery, and Management

Access Innovations, Inc.

Graph databases and the #panamapapers

darthvader42

Tendances (20)

Getting started in data science (4:3)

Getting started in Data Science (April 2017, Los Angeles)

Data Science Overview

Demystifying Data Science with an introduction to Machine Learning

Data Culture Series - Keynote & Panel - Birmingham - 8th April 2015

Building Satori: Web Data Extraction On Hadoop

What Is GDS and Neo4j’s GDS Library

David golynskiy resume it5

Introduction to Graph databases and Neo4j (by Stefan Armbruster)

Semantically Enabled Personal Information Management with Cluug.com

Graph Databases - Where Do We Do the Modeling Part?

Data Science: Harnessing Open Data for High Impact Solutions

AI in the Intelligent Workplace

Kurukshetra - Big Data

Personalized News and Video Recomendation System at LinkSure

Paving The Way To Data Driven

Smartlogic, Semaphore and Semantically Enhanced Search – For “Discovery”

Big Data Content Organization, Discovery, and Management

Graph databases and the #panamapapers

Similaire à Rapid Data Exploration With Hadoop

Building Competitive Moats With Data

Peter Skomoroch

Thinkful DC - Intro to Data Science

TJ Stalcup

"Semantic Integration Is What You Do Before The Deep Learning". dev.bg Machine Learning seminar, 13 May 2019. It's well known that 80\% of the effort of a data scientist is spent on data preparation. Semantic integration is arguably the best way to spend this effort more efficiently and to reuse it between tasks, projects and organizations. Knowledge Graphs (KG) and Linked Open Data (LOD) have become very popular recently. They are used by Google, Amazon, Bing, Samsung, Springer Nature, Microsoft Academic, AirBnb… and any large enterprise that would like to have a holistic (360 degree) view of its business. The Semantic Web (web 3.0) is a way to build a Giant Global Graph, just like the normal web is a Global Web of Documents. IEEE already talks about Big Data Semantics. We review the topic of KGs and their applicability to Machine Learning.

Semantics and Machine Learning

Vladimir Alexiev, PhD, PMP

Data Science-Why?What?How? By Hari Prasad

Hari Prasad

Hadoop and SAP BI

Praveen Kumar (Tyagi)

SDSC18 and DSATL Meetup March 2018

CareerBuilder.com

Data Foundation for Analytics Excellence by Tanimura, cathy from Okta

Tin Ho

2017 06-14-getting started with data science

Thinkful

Intro to Data Science

TJ Stalcup

How Oracle Uses CrowdFlower For Sentiment Analysis

CrowdFlower

Joe Caserta, President at Caserta Concepts addressed the challenges of Business Intelligence in the Big Data world at the Third Annual Great Lakes BI Summit in Detroit, MI on Thursday, March 26. His talk "Architecting for Big Data: Trends, Tips and Deployment Options," focused on how to supplement your data warehousing and business intelligence environments with big data technologies. For more information on this presentation or the services offered by Caserta Concepts, visit our website: http://casertaconcepts.com/.

Architecting for Big Data: Trends, Tips, and Deployment Options

Caserta

Ds01 data science

DotNetCampus

Big databigideasit4bc

Vincent Ohprecio

Frank Bien Opening Keynote - Join 2016

Looker

Frank Bien Opening Keynote - Join 2016

Looker

Big Data Landscape 2018

Leanne Hwee

Talk given at Big Boulder conference hosted by Gnip in Boulder, Colorodo on June 21, 2012. This talk provides an intro to Data Science at LinkedIn, and highlights the type of roles a Data Science team can play at a data-driven company. We use data (1) to create products that truly serve our members, (2) to derive insights, and (3) to generate wisdom which enables us to take the products and company to the next level. LinkedIn's data on 160+ million professionals' careers and networks provides a fascinating playground for data scientists to discover data insights about career trends, the social web and the economy.

Data Science at LinkedIn - Data-Driven Products & Insights

Yael Garten

David Bernstein of eQuest, the global leader in job-posting delivery and job board performance analytics, discusses how Big Data analysis provides organizations with greater recruitment marketing effectivenss than ever before. By not only delivering predictive information on job postings but by also taking a holistic look at your talent pipeline, Big Data analysis provides the insight organizations need to make better-informed decisions more quickly, reducing time-to-hire, costs and administrative burden.

Big Data for HR

David Bernstein

What Managers Need to Know about Data Science

Annie Flippo

Neo4j GraphTalk Düsseldorf - How Graphs revolutionise Identity & Access Manag...

Neo4j

Similaire à Rapid Data Exploration With Hadoop (20)

Building Competitive Moats With Data

Thinkful DC - Intro to Data Science

Semantics and Machine Learning

Data Science-Why?What?How? By Hari Prasad

Hadoop and SAP BI

SDSC18 and DSATL Meetup March 2018

Data Foundation for Analytics Excellence by Tanimura, cathy from Okta

2017 06-14-getting started with data science

Intro to Data Science

How Oracle Uses CrowdFlower For Sentiment Analysis

Architecting for Big Data: Trends, Tips, and Deployment Options

Ds01 data science

Big databigideasit4bc

Frank Bien Opening Keynote - Join 2016

Big Data Landscape 2018

Data Science at LinkedIn - Data-Driven Products & Insights

Big Data for HR

What Managers Need to Know about Data Science

Neo4j GraphTalk Düsseldorf - How Graphs revolutionise Identity & Access Manag...

Plus de Peter Skomoroch

Bridging the AI Gap: Building Stakeholder Support

Peter Skomoroch

Machine learning drove massive growth at consumer internet companies over the last decade, and this was enabled by open software, datasets, and AI research. For many problems, machine learning will produce better, faster, and more repeatable decisions at scale. Unfortunately, building and maintaining these systems is still extremely difficult and expensive. As more machine learning software moves to production, many of our traditional tools and best practices in software development will change. Pete Skomoroch walks you through what you need to know as we shift from a world of deterministic programs to systems that give unpredictable results on ever-changing training data. To navigate this world powered by nondeterministic data-dependent programs, we’ll also need a new development stack to help us write, test, deploy, and monitor machine learning software. Presented at OSCON Portland July 18, 2019

Managing Machines: The New AI Dev Stack

Peter Skomoroch

Companies that understand how to apply AI will scale and win their respective markets over the next decade. That said, delivering on this promise and managing machine learning projects is much harder than most people anticpate. Many organizations hire teams of PhDs and data scientists, then fail to ship products that move business metrics. The root cause is often a lack of product strategy for AI, or the failure to adapt their product development processes to the needs of machine learning systems. This talk will cover some of the common ways machine learning fails in practice, the tactical responsibilities of AI product managers, and how to approach product strategy for AI. Peter Skomoroch, former Head of Data Products at Workday and LinkedIn, will describe how you can navigate these challenges to ship metric moving AI products that matter to your business. Peter will provide practical advice on: * The role of an AI Product Manager * How to evaluate and prioritize your AI projects * The ways AI product management differs from traditional product management * Bridging the worlds of design and machine learning * Making trade offs between data quality and other business metrics

Product Management for AI

Peter Skomoroch

Companies that understand how to apply machine intelligence will scale and win their respective markets over the next decade. That said, delivering on this promise is much harder than most executives realize. Without large amounts of labeled training data, solving most AI problems isn’t possible. The talent and leadership to bridge the worlds of product design, machine learning research, and user experience are scarce. Many organizations will tackle the wrong problems and fail to ship successful AI products that matter to their customers. Pete Skomoroch explains how to navigate these challenges and build a business where every product interaction benefits from your investment in machine intelligence. This talk was presented at the 2019 Strata Data Conference in London. Topics include: Who defines the data vision and roadmap in your organization? Who is accountable for building and expanding your competitive moat? Investing in foundational data infrastructure, training, logging, and tools Fostering executive support for exploration and innovation, including user-facing data product and algorithm development How to evaluate new machine intelligence projects and develop a portfolio that delivers How AI product management differs from traditional product management How to bridge the worlds of design and machine learning to get to product-market fit Defining a framework for trading off investments in data quality, machine learning relevance, and other business objectives

Executive Briefing: Why managing machines is harder than you think

Peter Skomoroch

Examples, techniques, and lessons learned building data products over the last 4 years at LinkedIn. Pete Skomoroch is a Principal Data Scientist at LinkedIn where he leads a team focused on building data products leveraging LinkedIn's powerful identity and reputation data. The talk describes some techniques and best practices applied to develop products like LinkedIn Skills & Endorsements. This talk was presented at the SF Data Science Meetup on September 19th, 2013

SF Data Science: Developing Data Products

Peter Skomoroch

Skills, Reputation, and Search

Peter Skomoroch

Endorsements are a one-click system to recognize someone for their skills and expertise on LinkedIn, the largest professional online social network. This is one of the latest “data features” in LinkedIn’s portfolio, and the endorsement ecosystem generates a large graph of reputation signals and viral user activity. In this talk, we’ll examine the practical aspects of building a data feature like Endorsements. We’ll talk about marrying product design and data, deep diving into several of the lessons we’ve learned along the way - all using skills & endorsements as an empirical case study. We’ll include technical detail on our approaches and how we combine crowdsourcing, machine learning, and large scale distributed systems to recommend topics to users.

LinkedIn Endorsements: Reputation, Virality, and Social Tagging

Peter Skomoroch

Examples, techniques, and lessons learned building data products over the last 3 years at LinkedIn. Pete Skomoroch is a Principal Data Scientist at LinkedIn where he leads a team focused on building data products leveraging LinkedIn's powerful identity and reputation data. The talk describes some techniques and best practices applied to develop products like LinkedIn Skills & Endorsements. This was the inaugural UberData Tech Talk, held in SF at Uber HQ.

Developing Data Products

Peter Skomoroch

Practical Problem Solving with Data - Onlab Data Conference, Tokyo

Peter Skomoroch

Practical problem solving with data involves more than just visualization or applying the latest machine learning techniques. Intuition, domain knowledge, and reasonable approximations can mean the difference between a successful model and a catastrophic failure. We’ll dive into some best practices I’ve extracted from solving real world problems like computing trending topics, cleaning election data, and ranking experts on social networks. New analysts or engineers are often lost when textbook approaches fail on real world data. Drawing inspiration from problem solving techniques in mathematics and physics, we will walk through examples that illustrate how come up with creative solutions and solve problems with big data.

Street Fighting Data Science

Peter Skomoroch

Data Mashups -Data Science Summit

Peter Skomoroch

O'Reilly Where 2.0 2011 As a result of cheap storage and computing power, society is measuring and storing increasing amounts of information. It is now possible to efficiently crunch Petabytes of data with tools like Hadoop. In this O'Reilly Where 2.0 tutorial, Pete Skomoroch, Sr. Data Scientist at LinkedIn, gives an overview of spatial analytics and how you can use tools like Hadoop, Python, and Mechanical Turk to process location data and derive insights about cities and people. Topics: * Data Science & Geo Analytics * Useful Geo tools and Datasets * Hadoop, Pig, and Big Data * Cleaning Location Data with Mechanical Turk * Spatial Tweet Analytics with Hadoop & Python * Using Social Data to Understand Cities

Geo Analytics Tutorial - Where 2.0 2011

Peter Skomoroch

Prototyping Data Intensive Apps: TrendingTopics.org

Peter Skomoroch

Amazon EC2 may offer the possibility of high performance computing to programmers on a budget. Instead of building and maintaining a permanent Beowulf cluster, we can launch a cluster on-demand using Python and EC2. This talk will cover the basics involved in getting your own cluster running using Python, demonstrate how to run some large parallel computations using Python MPI wrappers, and show some initial results on cluster performance.

Elasticwulf Pycon Talk

Peter Skomoroch

Plus de Peter Skomoroch (14)

Bridging the AI Gap: Building Stakeholder Support

Managing Machines: The New AI Dev Stack

Product Management for AI

Executive Briefing: Why managing machines is harder than you think

SF Data Science: Developing Data Products

Skills, Reputation, and Search

LinkedIn Endorsements: Reputation, Virality, and Social Tagging

Developing Data Products

Practical Problem Solving with Data - Onlab Data Conference, Tokyo

Street Fighting Data Science

Data Mashups -Data Science Summit

Geo Analytics Tutorial - Where 2.0 2011

Prototyping Data Intensive Apps: TrendingTopics.org

Elasticwulf Pycon Talk

Dernier

Scaling API-first – The story of a global engineering organization

Radu Cotescu

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Civil Lines Women Seeking Men

Delhi Call girls

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

[2024]Digital Global Overview Report 2024 Meltwater.pdf

hans926745

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Delhi Call girls

A Call to Action for Generative AI in 2024

Results

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Sara Mae O’Brien Scott and Tatiana Baquero Cakici, Senior Consultants at Enterprise Knowledge (EK), presented “AI Fast Track to Search-Focused AI Solutions” at the Information Architecture Conference (IAC24) that took place on April 11, 2024 in Seattle, WA. In their presentation, O’Brien-Scott and Cakici focused on what Enterprise AI is, why it is important, and what it takes to empower organizations to get started on a search-based AI journey and stay on track. The presentation explored the complexities of enterprise search challenges and how IA principles can be leveraged to provide AI solutions through the use of a semantic layer. O’Brien-Scott and Cakici showcased a case study where a taxonomy, an ontology, and a knowledge graph were used to structure content at a healthcare workforce solutions organization, providing personalized content recommendations and increasing content findability. In this session, participants gained insights about the following: Most common types of AI categories and use cases; Recommended steps to design and implement taxonomies and ontologies, ensuring they evolve effectively and support the organization’s search objectives; Taxonomy and ontology design considerations and best practices; Real-world AI applications that illustrated the value of taxonomies, ontologies, and knowledge graphs; and Tools, roles, and skills to design and implement AI-powered search solutions.

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Enterprise Knowledge

Presentation on how to chat with PDF using ChatGPT code interpreter

naman860154

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

Slack Application Development 101 Slides

praypatel2

08448380779 Call Girls In Friends Colony Women Seeking Men

Delhi Call girls

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Heather Hedden, Senior Consultant at Enterprise Knowledge, presented “The Role of Taxonomy and Ontology in Semantic Layers” at a webinar hosted by Progress Semaphore on April 16, 2024. Taxonomies at their core enable effective tagging and retrieval of content, and combined with ontologies they extend to the management and understanding of related data. There are even greater benefits of taxonomies and ontologies to enhance your enterprise information architecture when applying them to a semantic layer. A survey by DBP-Institute found that enterprises using a semantic layer see their business outcomes improve by four times, while reducing their data and analytics costs. Extending taxonomies to a semantic layer can be a game-changing solution, allowing you to connect information silos, alleviate knowledge gaps, and derive new insights. Hedden, who specializes in taxonomy design and implementation, presented how the value of taxonomies shouldn’t reside in silos but be integrated with ontologies into a semantic layer. Learn about: - The essence and purpose of taxonomies and ontologies in information and knowledge management; - Advantages of semantic layers leveraging organizational taxonomies; and - Components and approaches to creating a semantic layer, including the integration of taxonomies and ontologies

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Enterprise Knowledge

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Explore 'The Codex of Business: Writing Software for Real-World Solutions,' a compelling SlideShare presentation that delves into digital transformation in healthcare. Discover through a detailed case study how Agile methodologies empower healthcare providers to develop, iterate, and refine digital solutions that address real-world challenges. Learn how strategic planning, user feedback, and continuous improvement drive success in deploying technologies that enhance patient care and operational efficiency. Ideal for healthcare professionals, IT specialists, and digital transformation advocates seeking actionable insights and practical examples of technology making a real difference.

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Malak Abu Hammad

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

Dernier (20)

Scaling API-first – The story of a global engineering organization

08448380779 Call Girls In Civil Lines Women Seeking Men

Tata AIG General Insurance Company - Insurer Innovation Award 2024

[2024]Digital Global Overview Report 2024 Meltwater.pdf

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

A Call to Action for Generative AI in 2024

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Boost PC performance: How more available memory can improve productivity

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Presentation on how to chat with PDF using ChatGPT code interpreter

Driving Behavioral Change for Information Management through Data-Driven Gree...

Slack Application Development 101 Slides

08448380779 Call Girls In Friends Colony Women Seeking Men

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Axa Assurance Maroc - Insurer Innovation Award 2024

The Codex of Business Writing Software for Real-World Solutions 2.pptx

What Are The Drone Anti-jamming Systems Technology?

GenCyber Cyber Security Day Presentation

Rapid Data Exploration With Hadoop

1. Rapid Data Exploration With Hadoop Peter Skomoroch Senior Data Scientist @peteskomoroch

2. Outline • Overview: LinkedIn Biz, Tech, & Analytics • Rapid Data Exploration 101 - Spatial Analytics Pig Code - Trend detection with Pig & Python - R Streaming Example • Deep Dive: Our Data Analysis Approach • Building Data Products • LinkedIn Data Insights

3. Connect the world’s professionals to make them more productive and successful

4. Professional Identity

5. LinkedIn at a glance • Founded in 2003 • #17 site in the US (Alexa) • 60+ million members • First million members = 477 days • Latest million = 9 days • 500K+ company proﬁles • 12+ million small business professionals • In 2009 - 1billion people searches • Average age: 41 • Household income $107,000 • 42% are “decision makers”

6. How International? • More than 50% international (members in over 200 countries & territories) • 13+ million in Europe • 4+ million in India • 3+ million in UK • #13 site in UK (Alexa)

7. How do we keep the lights on? • Proﬁtable since 2007 • Valued at over $1B at the last funding round • Subscriptions • Ads • Job Postings • Enterprise Client

8. Hadoop on LinkedIn 1,400+ members list “Hadoop” on their proﬁle What other skills do they have? •HBase, Lucene, Solr, MapReduce, Nutch... Where are they? Who do they work for? • 36% in Bay Area • 11% Yahoo! • 8% in India • 2% Apache Software Foundation • 6% in NYC • 1% LinkedIn • 4% in Seattle • 1% Google • 4% in Los Angeles • 1% Facebook

9. Hadoop at LinkedIn

10. Voldemort Data Storage Compact, compressed, binary data (something like Avro) Type can be any combination of int, double, float, String, Map, List, etc. => Sequence Files Example member definition: { ‘member_id’: ‘int32’, ‘first_name': 'string', ’last_name': ’string’, ‘age’ : ‘int32’ … }

11. Getting Data In •From Databases (user data, news, jobs etc.) • Need a way to get data reliably periodically • Need tests to verify data • Support for incremental replication • Solution: Transmogrify Driver Program • InputReader: JDBCReader, CSV Reader • Output Writer: JDBCWriter, HDFS writers • From web logs (page views, search, clicks etc) • Weblogs ﬁles are rsynced and loaded up in HDFS • Hadoop jobs for date cleaning and transformation.

12. Getting Data Out

13. Giving Back: Open Source http://sna-projects.com/sna/

14. Analytics Technologies

15. We Build Things With Data Give smart people great tools, enable them to solve problems

16. Prototyping Culture

17. How does Hadoop enable rapid data exploration?

18. Pig for Spatial Analytics

19. US County HeatMap

20. Pig for Trend Detection

21. Python Streaming Script

22. Sort Output & Display

23. R Streaming Also Easy *from http://www.stat.uiowa.edu/~luke/classes/295-hpc/

24. Let’s Talk Data

25. Business is recognizing the importance of analytics

26. What data do we start with?

27. We can also leverage... • Connection Graph • Company Pages • Recommendations • Talent Match • Address Book Uploads • Web Referrals • Search Logs • 1M+ Twitter Accounts • Proﬁle Views & Activity • Wikipedia Data • Job Postings • Mechanical Turk • LinkedIn Groups • Census, BLS, & Data.gov • LinkedIn Questions • Much more...

28. How do we think of Analytics? Data Jujitsu

29. Lots of Medium can be more powerful than Big >

30. Reconstruct Reality from Data Exhaust

31. Data Scientist Lessons • Follow the data, avoid assumptions • Sanity check the extremes (0, inﬁnity) • Don’t get mired in rare edge cases • Data Jujitsu: solve easier auxiliary problems • Build smaller consistent samples to test code • Establish a baseline model quickly, iterate often • Use the right tool for the job at hand • Iterate quickly with high level languages

32. Where did the bankers go?

33. We’re Hiring! http://sna-projects.com/sna/ pskomoro@linkedin.com @peteskomoroch

Rapid Data Exploration With Hadoop

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Rapid Data Exploration With Hadoop

Similaire à Rapid Data Exploration With Hadoop (20)

Plus de Peter Skomoroch

Plus de Peter Skomoroch (14)

Dernier

Dernier (20)

Rapid Data Exploration With Hadoop