2. Benchmarking
• Benchmarking is the process of continually searching for the best
methods, practices and processes, and either adopting or adapting
their good features and implementing them to become the “best of
the best.”
• How is benchmarking used?
• Compare performance of an existing process against other companies’
best-in-class practices.
• Determine how those companies achieve their performance levels.
• Improve internal performance levels.
Benchmarking is used for both comparison of performance
as well as to understand the potential for improvement
3. Benchmarking Compliance
Policy regarding benchmarking protocol should be communicated to all
employees involved, prior to contacting external organizations. Guidelines
should address the following areas:
Misrepresentation – do not misrepresent your identity in order to
gather information
Information requests – a request should be made only for
information your organization would be willing to share with another
company
Sensitive/proprietary information – avoid direct benchmarking of
sensitive or proprietary information
Confidentiality – treat all information as confidential
4. Benchmarking Compliance
Avoiding inappropriate communication and contacts
with competitors
Never propose, enter, or engage in a discussion
related to any agreements with a competitor to fix
prices, in terms or conditions of sale, costs, profit
margins, or other aspects of the competition
Keep communications with competitors to a minimum
– make sure there is a legitimate business reason for all
such communications
5. Benchmarking Compliance
Consult with business legal counsel before initiating any
contact with competitors regarding Six Sigma (or any other
topic):
Benchmarking
Training of suppliers and customers – they may also be competitors
6. Synchromesh Computing Services
Business Development
White Papers and Reports
Systematic Competitive
Intelligence Division (SCID)
Engineering and Marketing
Integration
Technical Evaluation of
Contracted Services
Sales Force Technical Training
with Joint Sales Calls
Zap Flyer Marketing System
Technical Communications
Fluent Core Documentation
System
• Product Evaluation
– Benchmarking and Certification
• Performance and
Power
– Software Testing
– Out of the Box Experience Testing
– First Impressions Comparison and
Competitive Analysis
• Software
Development
– Embedded Software Development
– Operating System Porting, Tuning
– Compiler Analysis
– System Security
7. History of the Company
ECL (EEMBC Certification Laboratory) formed February
1998
What they learned?
Industry-standard benchmark groups only go so far … still a pressing
need for head-to-head benchmarking, analysis
Must have a much wider range of benchmarks, and do “Customer
benchmarking”
Synchromesh Computing, LLC formed December 2003 to
provide much more comprehensive services by same
people
8. What They Do Well …
• Benchmarking and Performance Analysis
• Software Testing and Software Engineering
Consulting
• Software Development
• Industry Standards Group Creation,
Management, and Support
9. How They Benchmarked
• Obtain the system (hardware, software, documentation) and
create:
– Benchmark Test and Measurement Plan
– Benchmark Tests (software) and Test Fixtures
• Performance, Power Consumption, Code Size, Functionality
– Stabilize the Environment
– Execute Benchmark Tests, Control, and Measure
– Analyze, Debug, Assess, Compare/Contrast, Re-run
– Generate Test Results, Analysis, White Papers, etc.
– Consult
– If applicable, Certify Results
11. Examples of Key Business Processes
IBM Xerox British Telecom
Marketing Information Capture;
Marketing Selection;
Requirements;
Hardware/Software Development;
Service Development;
Production;
Customer Fulfillment/ Relationship;
Service Customer Feedback;
Marketing;
Solution Integration;
Financial Analysis;
Plan Integration;
Accounting;
Human Resources
IT Infrastructure
Customer Engagement;
Inventory Mgt. & Logistics;
Product Design / Engineering;
Product Maintenance;
Technology Maintenance;
Production & Operations Mgt.
Marketing Management;
Supplier Management;
Information Management;
Business Management;
Human Resources Management;
Leased & Capital Asset Mgt.
Legal;
Financial Management.
Direct Business;
Plan Business;
Develop Processes;
Manage Process Operations;
Provide Personnel Support;
Market Products & Services;
Provide Customer Service;
Manage Products & Services;
Provide Consultancy Services;
Plan the Network;
Operate the Network;
Provide Support Services;
Manage Information Resource;
Manage Finance;
Provide Technical R&D
Editor's Notes
Comply with GECS guidelines for:
Documenting competitor contacts – check with legal counsel first.
Participating in trade associations and standards - setting groups
Product Evaluation
Benchmarking and Certification
Performance and Power
Software Testing
Out of the Box Experience Testing
First Impressions Comparison and Competitive Analysis
Software Development
Embedded Software Development
Operating System Porting, Tuning
Compiler Analysis
System Security
Business Development
White Papers and Reports
Systematic Competitive Intelligence Division (SCID)
Engineering and Marketing Integration
Technical Evaluation of Contracted Services
FAE and Sales Force Technical Training with Joint Sales Calls
Zap Flyer Marketing System
Technical Communications
Fluent Core Documentation System
ECL (EEMBC Certification Laboratory) formed February 1998
Certifies benchmark scores for EEMBC (industry consortium)
Has certified literally hundreds of benchmark scores
Developed nearly all of the EEMBC benchmark source code (millions of lines of very portable C source code)
Gained experience with hundreds of software tool chains, hardware situations
What they learned?
Industry-standard benchmark groups only go so far … still a pressing need for head-to-head benchmarking, analysis
Must have a much wider range of benchmarks, and do “Customer benchmarking”
Synchromesh Computing, LLC formed December 2003 to provide much more comprehensive services by same people
Profitable since Day 1, no debt, growing each year
In March 2006 Synchromesh Computing bought ECL
They’ve been doing this for over 8 years ….
Benchmarking and Performance Analysis
Certified literally hundreds of benchmark scores for EEMBC
Developed nearly all of EEMBC source code (3 million lines)
Have been involved with SPEC, BAPCo, other groups
Consultants to large and small companies on performance, power: Freescale, AMD, SigmaTel, AMCC, Legerity, many others
Software Testing and Software Engineering Consulting
Trained hundreds of Engineers in Software Inspections and Software Engineering for companies
First Impressions ™ and Market Ready Validation ™ Programs are considered standards for software testing and evaluation
Freescale Semiconductor, Mentor Graphics, Green Hills Software, Texas Instruments, Symantec (Veritas), dozens of others
Software Development
Compilers, Operating Systems, Middleware, Applications
Industry Standards Group Creation, Management, and Support
As EEMBC Certification Laboratory (ECL), helped Found EEMBC (industry-standard benchmarking organization)
Provided all Technical Support (phone, email, web, etc.)
Provided all software development services
Chaired Technical Advisory Group for 5 years
Certified benchmark scores
Providing technical assistance to new RuBee data radio standard for the IEEE on behalf of Visible Assets, Inc.
We will be their Certification and Compliance Laboratory
We helped Found the SPEC-MBC (Multimedia Benchmark Committee) – also served as first Vice-Chairman
Exploring work with SI2 (Silicon Integration Initiative) consortium.