Amit Sheth, SEMANTIC CONTENT MANAGEMENT FOR ENTERPRISES AND NATIONAL SECURITY, Keynote at: CONTENT- AND SEMANTIC-BASED INFORMATION RETRIEVAL @ SCI 2002.

1. SEMANTIC CONTENT MANAGEMENT FOR ENTERPRISES AND NATIONAL SECURITY Amit Sheth CTO, Voquette*, Inc. Large Scale Distributed Information Systems (LSDIS) Lab University Of Georgia; http://lsdis.cs.uga.edu *Now Semagix, http://www.semagix.com July 15, 2002 © Amit Sheth Keynote CONTENT- AND SEMANTIC-BASED INFORMATION RETRIEVAL @ SCI 2002

7. Digital Content and Semantics

8. Central Role of Metadata Applications Back End "A Web content repository without metadata is like a library without an index." - Jack Jia, IWOV “ Metadata increases content value in each step of content value chain.” Amit Sheth Where is the content? Whose is it? Produce Aggregate What is this content about? Catalog/ Index What other content is it related to? Integrate Syndicate What is the right content for this user? Personalize What is the best way to monetize this interaction? Interactive Marketing Broadcast, Wireline, Wireless, Interactive TV Semantic Metadata

9. A Metadata Classification Data (Heterogeneous Types/Media) Content Independent Metadata (creation-date, location, type-of-sensor...) Content Dependent Metadata (size, max colors, rows, columns...) Direct Content Based Metadata (inverted lists, document vectors, LSI) Domain Independent (structural) Metadata (C++ class-subclass relationships, HTML/SGML Document Type Definitions, C program structure...) Domain Specific Metadata area, population (Census), land-cover, relief (GIS),metadata concept descriptions from ontologies Ontologies Classifications Domain Models User More Semantics for Relevance to tackle Information Overload!!

11. SCORE Architecture

12. SCORE Architecture Distributed agents that automatically extract relevant semantic metadata from structured and unstructured content Fast main-memory based query engine with APIs and XML output CACS provides automatic classification (w.r.t. WorldModel) from unstructured text and extracts contextually relevant metadata Distributed agents that automatically extract/mine knowledge from trusted sources Toolkit to design and maintain the Knowledgebase Knowledgebase represents the real-world instantiation (entities and relationships) of the WorldModel WorldModel specifies enterprise’s normalized view of information (ontology)

13. Voquette Enterprise Semantic Platform Product Components World Model WM Toolkit Knowledgebase and Metabase Main Memory Index XML APIs Web Services Enterprise Applications EA EA EA Semantic Engine Search Alerts Portals Directory Personalize Enhancement Engine CA CA CA Content Agent Monitor Content Agents Databases XML/Feeds Websites Email Content Sources Entity Extraction, Enhanced Metadata, Domain Experts Automatic Classification Classification Committee Reports Documents Structured Semi- Structured Unstructured CA Toolkit Knowledge Agent Monitor KS KS KS KS KA KA KA Knowledge Sources Knowledge Agents KA Toolkit Knowledgebase KB Toolkit Knowledge Agent Monitor KS KS KS KS KA KA KA Knowledge Sources Knowledge Agents KA Toolkit Metabase Enhancement Engine CA CA CA Content Agent Monitor Content Agents Databases XML/Feeds Websites Email Content Sources Entity Extraction, Enhanced Metadata, Domain Experts Automatic Classification Classification Committee Reports Documents Structured Semi- Structured Unstructured CA Toolkit

14. PERSON (OFAC, FBI, DPL) -politician (OFAC, FBI, CIA, CA) politician associated with politicalOrganziation politician held politicalOffice politician associated with politicalOffice -terrorist (OFAC, FBI, DPL) terrorist memberOf organization terrorist appears on watchList -companyExecutive (MG) companyExecutive holdsOffice companyPosition person has permanent address address (OFAC, FBI) person has dob(date of birth) (OFAC, FBI) person has pob(place of birth) (OFAC, FBI) Knowledge Sources Used THING -event (ICT) terroristOrganization participated in terroristSponsoredEvent (ICT) -politicalOffice (CIA, CA) politicalOffice office(s) within govtOrganization politicalOffice associated with organization -watchList (OFAC, FBI, DPL) terroristOrganization appears on watchList (OFAC, FBI, DPL) -organization (OFAC, FBI, FAS, ICT, CA, CIA) organization appears on watchList organization memberOf suborganization -company company manufactures product (ZD) company identifiedBy tickeySymbol (H) companyposition position in company (MG) company memberOf industry (H) -tickerSymbol (H) tickerSymbol memberOf exchange (H) PLACE -organization located in place (H, OFAC) -religiousAffiliation practiced in place (CIA) -company headquarters in city (H) Entity Classes and Relationships populated by these knowledge sources: JIVA Market Guide (MG) ZDNet (ZD) Hoover’s (H) Data supplied from NASA (DPL) Federation of American Scientists (FAS) C entral Intelligence Agency (CIA) The Interdisciplinary Center (ICT) Federal Bureau of Investigation (FBI) Capital Advantage (CA) Office of Foreign Assets Control ( OFAC)

17. Performance > 10,000 entities/relationships per hr. Population/update rate in a Knowledgebase with 1 million entities/relationships 1 minute (near real-time) Incremental Index Update Frequency 65ms Query Response Time (64 concurrent users) 1 - 10 ms Query Response Time (light load) > 1,980,000 Queries per server per hour

18. Information Extraction for Metadata Creation METADATA EXTRACTORS Key challenge: Create/extract as much (semantics) metadata automatically as possible WWW, Enterprise Repositories Digital Maps Nexis UPI AP Feeds/ Documents Digital Audios Data Stores Digital Videos Digital Images . . . . . . . . .

19. Video with Editorialized Text on the Web Automatic Categorization & Metadata Tagging (Web page) Auto Categorization Semantic Metadata

20. Extraction Agent Web Page Enhanced Metadata Asset Content Extraction and Knowledgebase Enhancement

21. Content Enhancement Workflow Semantic Metadata Syntax Metadata

22. Content Asset Index Evolution Extractor Agent for Bloomberg Scans text for analysis Metadata extracted automatically Asset Syntax Metadata Producer: BusinessWire Source: Bloomberg Date: Sept. 10 2001 Location: San Jose, CA URL: http://bloomberg.com/1.htm Media: Text Semantic Metadata Company: Cisco Systems, Inc. Creates asset (index) out of extracted metadata Asset Syntax Metadata Producer: BusinessWire Source: Bloomberg Date: Sept. 10 2001 Location: San Jose, CA URL: http://bloomberg.com/1.htm Media: Text Semantic Metadata Company: Cisco Systems, Inc. Topic: Company News Categorization & Auto-Cataloging System (CACS) Scans text for analysis Classifies document into pre-defined category/topic Appends topic metadata to asset Cisco Systems CSCO NASDAQ Company Ticker Exchange Industry Sector Executives John Chambers Telecomm. Computer Hardware Competition Nortel Networks Knowledge Base CEO of Competes with Syntax Metadata Asset Producer: BusinessWire Source: Bloomberg Date: Sept. 10 2001 Location: San Jose, CA URL: http://bloomberg.com/1.htm Media: Text Semantic Metadata Company: Cisco Systems, Inc. Topic: Company News Ticker: CSCO Exchange: NASDAQ Industry: Telecomm. Sector: Computer Hardware Executive: John Chambers Competition: Nortel Networks Headquarters: San Jose, CA Leverages knowledge to enhance metatagging Enhanced Content Asset Indexed Headquarters San Jose XML Feed Semantic Engine

23. Content which does contain the words the user asked for Extractor Agents Content which does not contain the words the user asked for, but is about what he asked for. Value-added Metadata Content the user did not think to ask for , but which he needs to know . Semantic Associations + + Intelligent Content End-User Intelligent Content Empowers the User

24. Example 1 – Snapshots (“Jamal Anderson”) Click on first result for Jamal Anderson View metadata. Note that Team name and League name are also included in the metadata Search for ‘Jamal Anderson’ in ‘Football’ View the original source HTML page. Verify that the source page contains no mention of Team name and League name . They are value-additions to the metadata to facilitate easier search.

25. Semantic Application Example – Research Dashboard Focused relevant content organized by topic ( semantic categorization ) Automatic Content Aggregation from multiple content providers and feeds Related relevant content not explicitly asked for (semantic associations) Competitive research inferred automatically Automatic 3 rd party content integration

26. Internal Source 1 Research Internal Source 2 External feeds/Web (e.g. Reuters) Voquette Metabase World Model Third-party Content Mgmt And Syndication Semantic Engine 1 2 3 4 Cisco story from Source 1 passed on to add semantic associations Consults Knowledge Base for Cisco ’s competition Returns result: Lucent is a competitor of Cisco Lucent story from external feeds picked for publishing as “semantically related” to Cisco story – passed on to Dashboard Story on Lucent Story on Cisco XCM-compliant metadata, XML or other format Semantic Application ASP/Enterprise hosted Extractor Agent 1 Extractor Agent 2 Extractor Agent 3 Metadata centric Content Management Architecture

27. Related Stock News Semantic Web – Intelligent Content Industry News Technology Products COMPANY EPA Regulations Competition COMPANIES in Same or Related INDUSTRY COMPANIES in INDUSTRY with Competing PRODUCTS Impacting INDUSTRY or Filed By COMPANY Important to INDUSTRY or COMPANY Intelligent Content = What You Asked for + What you need to know! SEC

28. led by Same entity Human-assisted inference Knowledge-based & Manual Associations Syntax Metadata Semantic Metadata

29. Blended Semantic Browsing and Querying (Intelligence Analyst Workbench)

31. Visionics AcSys Security Portal Check-in Interrogation Boarding Gate Airport Airspace Voquette Knowledgebase Metabase Threat Scoring Gov’t Watchlists News Media Web Info LexisNexis RiskWise Passenger Records Reservation Data Airline Data Airport Data Airline and Airport Data Future and Current Risks Airport LEO ARC AvSec Manager Data Management Data Mining IPG

32. Sources Used Content Sources : Africa News Service AFX News – Asia/UK/Europe AP Worldstream Asia Pulse BusinessWire ComputerWire (CTW) EFE News Services FWN Select Itar-TASS Knight Ridder News (Open) Knight-Ridder Open M2 - International M2 Airline Industry Information New World Publishing PR Newswire PRLine (PRL) Resource News International RosBusiness United Press International UPI Spotlights Knowledge Sources: FBI - Most Wanted Terrorists Denied Persons Lists Terrorism Files ICT Office of Foreign Asset Control (OFAC) Hamas terrorists CNN Locations FAA_Airport_Codes About.com Comtex_International Hindustan Times JerusalemPost CNN Newstrove_Hamas

33. Voquette’s Semantic Technology enables flight authorities to : - take a quick look at the passenger’s history - check quickly if the passenger is on any official watchlist - interpret and understand passenger’s links to other organizations (possibly terrorist) - verify if the passenger has boarded the flight from a “high risk” region - verify if the passenger originally belongs to a “high risk” region - check if the passenger’s name has been mentioned in any news article along with the name of a known bad guy Interrogation Kiosk – Unique Advantages of Voquette Smith John

34. Threat Score Components Smith John WATCHLIST ANALYSIS Action : Voquette’s rich knowledgebase is automatically searched for the possible appearance of this name on any of the watchlists Ability Proven : Ability to automatically aggregate relevant rich domain knowledge and automatically co-relate it and rank the threat factors to indicate threat level of the passenger on the watchlist front METABASE SEARCH Action : Voquette’s rich metabase is searched for this name and associated content stories mentioning the passenger’s name are retrieved Ability Proven : Ability to automatically aggregate and retrieve relevant content stories, field reports, etc. about the passenger that can be used by flight officials to determine if the passenger has any connections with known bad people or organizations appearsOn watchList : FBI KNOWLEDGEBASE SEARCH Action : Voquette’s rich knowledgebase is searched for this name and associated information like position, aliases, relationships (past or present) of this name to other organizations, watchlists, country, etc. are retrieved Ability Proven : Ability to automatically aggregate relevant rich domain knowledge about a passenger and automatically co-relate it with other data in the knowledgebase to present a visual association picture to the flight official LEXIS NEXIS ANNOTATION Action : Information about or related to the passenger returned by Lexis Nexis is enhanced by linking important entities to Voquette’s rich knowledgebase Ability Proven : Ability to automatically aggregate relevant rich domain knowledge, recognize entities in a piece of text and further automatically co-relate it with other data in the knowledgebase to present a clear picture about the passenger to the flight official Flight Coutry Check 45 0.15 Person Country Check 25 0.15 Nested Organizations Check 75 0.8 Aggregate Link Analysis Score: 17.7 LINK ANALYSIS Action : Semantic analysis of the various components (watchlist, Lexis Nexis, knowledgebase search, metabase search, etc.) to come up with an aggregate threat score for the passenger Ability Proven : Ability to automatically aggregate relevant rich domain knowledge, recognize entities in a piece of text, automatically co-relate it with other data in the knowledgebase, search for relevant content to present an overall idea of the threat level fo the passenger, allowing him to take quick action

35. Query Comparison: Voquette vs. RDBMS

36. JIVA Semantic Console Start-up Interface The mission of the JIVA project is to gather and analyze as much information of diverse kinds about suspected individuals, terrorist and other groups, organizations, events, etc. For this Terrorism domain, the JIVA Semantic Console provides an information retrieval interface (shown below) that displays some fundamental semantic attributes (based on a corresponding Terrorism domain model) to enable information retrieval in the right context. Most fundamental semantic attributes specific to the Terrorism domain (fully customizable) Syntactic or domain-independent attributes for general and media-specific search Analyst can enter search values in the appropriate attribute fields (to search in the right context) Analyst can choose the type of media of the desired content Once all other values are set, click the “ Search” button to search semantically Search interface with more search features (explained later) JIVA Functionality Interface

38. Facilitating Knowledge Discovery On clicking any bin Laden-related entity (e.g. Al Qaeda), a page is displayed to the analyst showing knowledge pertaining to that entity, which can be used in a BSBQ mode, as described on the previous screen. Continuing this integrated approach of Semantic Browsing and Querying, the analyst has the necessary ammunition to perform Knowledge Discovery . The analyst can follow his train of thought as he browses and queries to possibly discover unexpected relationships and links between entities at various levels in an indirect manner. Automatically uncovering such hidden related entities facilitates addition of new and meaningful entities and relationships to the analyst’s assessment tasks. JIVA

39. Wireless Application of Semantic Metadata and Automatic Content Enrichment  Clicking on the link for Cisco Analyst Calls displays a listing sorted by date. Semantic filtering uses just the right metadata to meet screen and other constrains. E.g., Analyst Call focuses on the source and analyst name or company. The icon denote additional metadata, such as “Strong Buy” by H&Q Analyst. MyStocks News Sports Music MyMedia    $  My Stocks CSCO NT IBM Market CSCO Analyst Call Conf Call Earnings    11/08 ON24 Payne 11/07 ON24 H&Q  11/06 CBS Langlesis CSCO Analysis

41. Metadata for Automatic Content Enrichment Interactive Television This segment has embedded or referenced metadata that is used by personalization application to show only the stocks that user is interested in. This screen is customizable with interactivity feature using metadata such as whether there is a new Conference Call video on CSCO. Part of the screen can be automatically customized to show conference call specific information– including transcript, participation, etc. all of which are relevant metadata Conference Call itself can have embedded metadata to support personalization and interactivity.

43. Metadata Usage: Keyword, Attribute and Content Based Access The VisualHarness system at LSDIS/UGA