Simple steps and tips to improve IT infrastructure operations #yapcasia #yapcasiaE

Simple steps and tips to
improve IT infrastructure
operations
2015-08-22 15:30-16:30
@ YAPC::Asia Tokyo 2015 (Day 2)
Yuichiro Saito (@koemu)
© Yuichiro Saito (@koemu), 2015 1

Information
• Feel free to take photos, tweet, and blog
about this talk !
• Twitter hashtag: #yapcasiaE
• Slides will be uploaded to slideshare.

About Me

Yuichiro Saito (@koemu)
• Software Engineer @ HEARTBEATS Corp.
(MSP)
• Specialty: Improve Engineering Productivity
• Work Experience: Stock analysis system
development, CMS, EC, NLP, Smartphone
game development, AR

Now on SALE !
Webアプリエンジニア養成読本

Question

What are you ?
• Software Engineer ?
• Operation Engineer ?
• Project Manager ?
• Other ?

TOC
• 0: Background
• 1: People WANT to talk about their problems
• 2: Involve closest person
• 3: Evaluating performance and beyond

0: Background

About HEARTBEATS (HB)
• MSP (Managed Service Provider1
)
• 24/7 manned monitoring.
• Stable, secure, customer-centric service.
1
MSPAlliance, “Deﬁnition of Managed Services”, http://www.mspalliance.com/blog/
definition-of-managed-services/ (Web)

In 2012
• External
• Raise of IaaS (AWS, etc ...)
• Easier to handle service instances with IaaS
(compared to on-premise).
• Internal: Environment within HB
• Manual operations.
• Oh, It's archaic :-(

1: People WANT to talk about their problems

Changes
• More instances to operate than on-premise
era.
• Able to handle instances with software.

Problems
• More overtime work
• Manual operations in the IaaS era !
• Decreasing efﬁciency
• Slow lead time.
• Delayed adoption of new technologies.

Wow, this is sucks !
• Miss out on business opportunities.
• We must abandon our manual operations, or
else we will be left behind in the IaaS era.

We need improvement
• But... improvement is inseparable from
resistance!
• But we still need to solve our problems.
• If I can make life easier for people, they will
be more susceptible to improvement.
• DeMarco said: "People can't embrace
change unless they feel safe."2
2
Tom DeMarco, “The Deadline: A Novel About Project Management”, Dorset House, 1997

Understanding the problem
• Find Hints
• We don't have comprehensive information.
• Identify "goal-reality" pairs3
• e.g.) Management principle (goal) -
Efﬁciency (reality)
3
Eric G. Flamholtz, Yvonne Randle, “Growing Pains: Transitioning from an Entrepreneurship to a
Professionally Managed Firm”, Jossey-Bass, 2000

Expectation
• "Inner Work Life"4
• Performance is affected by motivation.
• Motivation is affected by happiness.
4
Amabile, T.M. and Kramer, S.J., “Inner work life: understanding the subtext of business
performance”, Harvard Business Review, pp. 72-83, May 2007

Research
• Interview
• Questionnaire

Two types of difficult tasks
1. Short time commitment and many
repetitions.
2. Infrequent but long time commitment.

Short time and TOO MANY repeat
Cause
• Inefﬁcient manual operations.
• Operating environment is not automated.

Few repeats but LONG time
Cause
• Requires manual operation, lack of sharing of
information.
• The replacement personnel runs into
unexpected trouble due to lack of
communication during transition.

Analysis parameters
• Two factor
• Difﬁculty
• COST

What's HARD ?
• Mental stress
• Workload
• Short lead time
• Dependent on individual skills
• Technical unfeasible

What's COST
• Preparation cost
• Development cost
• Introduction, training cost
• Cost impact to the organization

Our conclusion so far
1. People will be willing to change if they
understand that it will make their life easier.
2. Identify what is making their life difﬁcult.
3. Prioritize according to difﬁculty and cost.

2: Involve closest person

Case study: hb-acns

Developed internal product
• hb-agent5
: Automatic server building tool
• hb-gendoc: Automatic document building
tool
• Cacti bulk conﬁguration tool
• hb-acns6
: Our topic !
6
Yuichiro Saito, “hb-acns - サーバ監視・メトリック取得設定の自動化システムのご紹介 - インフラ
エンジニアway”, 2015, http://heartbeats.jp/hbblog/2015/06/hb-acns.html (blog)
5
Yuichiro Saito, “hb-agent - 構築・監視項目検出自動化ツール hb-agentのご紹介 - インフラエンジ
ニアway”, 2015, http://heartbeats.jp/hbblog/2015/05/hb-agent.html (blog)

What's hb-acns ?
• Automatic register to monitoring system
(Nagios, Cacti).
• Free ALL engineers from the pains of manual
conﬁguration.
• Preconﬁgureed rules can be converted into
scripts.

Why ?
• Impossible to manually set up monitoring
settings for all IaaS instances.

Two targets
• Small terms with many projects.
• Highest need for efﬁciency.
• Big terms with few projects.
• Easy standardization.

Why don't use OSS ?
• We have:
• Multiple datacenters.
• Many customers with varying needs.
• An established operation workﬂow to meet
these need.
• The existing OSS was unsuitable.

Opportunity
• We ware about to start a new project.
• The CTO and I decided that this would be a
pilot project.

Plan of pilot project
1. Plan: Speciﬁcations, Preparation
2. Try: Develop, Test to use, Find Problem
3. Feedback: Interview, Questionnaire
4. Spread: Propagate, Improve

Tips of pilot project
1. Get CTO support.
2. Introduce merit.
3. Troubleshoot immediately.

Get CTO support
• Initiative by CTO.
• Endorsement from CTO.
• Emphasize exective leadership.

Introduce merit.
• Everyone is busy => Resistant to change
• Introduce merit, repeat and repeat.
• What is improve task ?
• Which of my jobs wolud be improved ?
• Let people know that it can be done !

Troubleshoot immediately
• There will always be unexpected trouble.
• If the response is slow, people will stop trying.

Propagate
• Documentation
• Can use him/her self DIY solutions.
• Hands-on demonstrations7
• Solve the fears of weakness DIY solutions.
• Spread to nearly all of staff (over 40) :-)
7
Yuichiro Saito, “ツール普及のために社内ハンズオンに取り組んでみた - インフラエンジニアway”,
2014, http://heartbeats.jp/hbblog/2014/11/handson.html (blog)

(hb-acns chat ops)
• That is executed every day.

• Plan -> Try -> Feedback -> Spread
• Get CTO support
• Hands-on workshops lower entry barriers.

3: Evaluating performance and beyond

3 weeks• Period of development.

3 months
• The beggining of the pilot project to the
hands-on workshops.

1/10 times
• Lead time of monitoring setup.

Workload time reduced >>
Time invested in development

Cost performance
• Increase proﬁtability
• Reduce monitoring setup and update time.
• Increase proﬁt
• We can get more customer by saving time !

Side effects
• People understood that ...
• Programming will increase efﬁciency.
• They can write code, too !8
8
Ryota Yoshikawa (@rrreeeyyy), “Infrastructure as Code の始め方 // Speaker Deck”, Open Source
Conference 2014 Tokyo/Fall, 2014, https://speakerdeck.com/rrreeeyyy/osc-20141018-
infra-as-code (Slide)

Future goals
• Increase operation efﬁciency through
programming
• More in-depth study of programming.
• Solve problems by programming.
• Basis formed by operating skills (Tuning,
Middleware, Clarify the problem, etc ...)

• Succeeded in improving operation
productivity.
• People recognized the need of programming
to improve operations.
• Programming improves operations efﬁcient.

Conclusion

Don't be afraid to change !
• People will accept growing pains if it will
make their lives easier.
• Hands-on workshops are great for reducing
barriers and anxiety.
• Programming improves operations efﬁcient.

END

Simple steps and tips to improve IT infrastructure operations #yapcasia #yapcasiaE

Recommandé

Recommandé

Contenu connexe

Similaire à Simple steps and tips to improve IT infrastructure operations #yapcasia #yapcasiaE

Similaire à Simple steps and tips to improve IT infrastructure operations #yapcasia #yapcasiaE (20)

Plus de Yuichiro Saito

Plus de Yuichiro Saito (15)

Dernier

Dernier (20)

Simple steps and tips to improve IT infrastructure operations #yapcasia #yapcasiaE