QCon Tokyo 2014
QCon2014 My Report Top
Detail Report#1/8
Keynote#1 Velocityの好循環 早く進むことの重要性 Google & eBayで学んだこと
Google Blame-Free Postmortem
- Postmortem,,, (後日スライドがアップされたら補足)
Action Item
How will we change process, technology, documentation
How Could we have automated the problems away?
How could we have diagnosed mode quickly
How could we have resorted service more quickly
Follow up!
Virtuous cycle(好循環) of improvement
Results -> Honesty - Learn -> Improve
Organization : Service Teams
Small focused teams
- single service or set of related services
- Minimal, well-defined "interface"
Clear contract between teams
- Functionality
- Service levels and performance
Google Services
All engineering groups organized into "service"
- Gmail, App Engine, Bigtable, etc.
Self-sufficient and autonomous
Layered on One another (階層化されている)
Result
Very small teams achieve great things
Organization: Ownership culture
Give teams autonomy
- Freedom to choose technology, methodology, working environment
- Responsibility for the result of those choices
Hold them accountable for results
- Give a team a goal, not a solution
Example: KIXEYE product (which is Service Chassis)
Goal Produce a chassis for building scalable game services
Minimal resources, minimal direction
- 3 people
- consider building with OSS ### Results
- exceeded expectations: chassis , transport, service, service,template auto-scaled,
- 15 minutest from no code
- making them to OSS
- Results -> Autonomy-> 組織化
Organization: Collaboration
- One team across engineering, product , operation
Google Co-location & Multiple organizations
- Engineering
- Product
- Operations
- Support
- Different reporting structures to different VPs
Virtual team with single Goal
- All work to make Google App Engine Successful
- Coworkers are "Us", not "Them"
- Never occurred to us that other organizations were not "our team"
Process: Experimentation
Engineer successes
- constant iteration
- Launch is only the first step
- A/B Testing needs to be a core competence
Many small experiments sum to big wins
eBay machine-learned Ranking
Ranking function for search results
* which item should appear 1st, 10th, 100th, 1000th
* Before: small number of hand-tuned factors
* Goal Thousands of factors
Experimentation proces
- Predictive models: query->view, view->purchase, etc
- Hundreads of parallel A/B Tests
- Fully year of steady, incremental improvements
Result: 2% increase in eBay revenue (1.2 billion US$ / 60 billion)
* Experiment -> Learn ->Impove -> Results
Process: Quality Discipline
"Quality is a Priority-0 feature"
Automated tests help you go faster
- Tests have your back
- Confidence to break things refactor mercilessly (既存物を壊して容赦なくリファクタする)
- Catch bugs earlier, fail faster
Faster to run on solid ground than on quicksand
Process: Institutionalize Quality
Development Practices
- code review
- continuous testing
- continuous integration
Quality Automation
- Automated testing frameworks
- canary releases to production (製品の試験リリース)
(100台のうち１台を新しいバージョンにかえて、もんだいないことを確認してから他を新しいものに変える)
Google Engineering Discipline
Solid Development practices
- code reviews before submission
- automated tests for everything
- Single logical source repository
Result: Internal open source model
- Not "here is a bug report"
- instead " here is the bug; here are the code hanges: here is the test that verifies the changes" (「これはバグで、これはコード変更で、これはその変更を検証するテストです」)
Cycle of Quality
- Engineering Discipline -> Solid Foundation -> Faster and Better -> Results
Process: Manage technical Debt(技術的負債の管理)
Make Explicit Tradeoffs
- triangle: date vs quality vs features
- when you choose date and features, you implicity choose a level of quality (多くの機能と早い日付を選んだら、低い品質も選んだことになる)
Manage your debt
- Plan how and when you will pay it off
- maintain a sustainable level of debt
Don't have time to do it right
*Wrong don't have time to do it twice!
(間違えることで２度やってしまうことの方が問題)
Vicious cycle of Technical debt(悪いサイクル)
- Quick-And Dirty -> Technical Debt -> No time to do it right
Virtuous cycle of Technical debt (いいサイクル)
- Invest -> Solid foundation - Faster and better -> Results
People: Hire and retain the best
Hire 'A' Players
- In creative disciplines, top performers are 10x more productive
Confidence(信頼の伝搬)
- A players bring A players
- B players bring C players (チョー厳しいことをいってます。 トップレベルでないと下がる一方か、、、)
Google Hiring
Goal Only hire top talent
- False negatives are OK: false positives are not !
- Famously challenging interviews
- Very detailed interviewer feedback
- Hiring committee decides whether to hire
- Separately assign person to group
People: Respect People
People are not interchangeable
* Different skills, interests, capabilities
* Create a Symphony, not a factory
Most valuable and irreplaceable asset
* treat people with care and respect
BadExample: eBay "Train Seats"
some years ago, eBay's development process
* Design and estimate project
("train seat" == 2 engineer - weeks)
* Assign engineers from common pool to implement tasks
* Designer ndoes not implemnt : implementers do not design
many problems
- no pride of ownership
- no long -term ownership
Virtuous cycle of people
Hire A players -> treat well -> keep and retain -> results
Q&A
small teamのところでは、Functional, respectのところではsymphony not factoryといっていて微妙に違うことをいっている気がしますが。
いい質問です。
A players 3-4いて、それぞれが異なる知識をもっていれば、十分なアウトプットを出せるのでそれがベストなチームの形。
スタートアップフェーズはチーム内で完結する形でシンフォニーをつくる。組織やプロジェクトが大きくなるにつれてFunctionalなスモールのチームを構成して、チーム間でシンフォニーをつくるようにする。
Triangleで Feature set Quality Date といっていたがcostが抜けているのでは？
Resource(Cost)はその3つの上位に来る概念。 Resourceがたくさんあればより大きなことをできる。
