AWS re:Invent 2019 ノート

AWS re:Invent 2019 に参加したまとめ。

感想

設計について blast radius というキーワードが目についたので個人的に頭に入れる必要がある。

キーとして train your builders というものがあり、学習型カンファレンスである re:invent 自体もそうだし、 builders' library によるベストプラクティスの提供、DeepRacer のような機械学習に対する開発者の学習サービスの提供など、aws がその機会を提供しているのも各所で感じられた。

動向としては新サービスとしては ai/ml 系、on-prem との障壁を無くすために高性能な処理と低遅延のネットワーク系の追加が主であるように見えた。

イベントについてメモ

las vegas のホテル＆カジノ群で行われる大規模なイベントという感じで、隣の会場まで歩いて 20 min くらい掛かるので歩き疲れた（Bally's という交通の便が最適ではないホテルに滞在したので顕著だった）。 可能なら会場となるホテルのどれかに滞在した方がシャトルでの移動で楽になりそう。

食事は朝昼で buffet か grab & go、食べるところは大抵広かったので初日の昼だけ grab & go にしたけど後は buffet で食べた。料理はアジア料理、メキシコ料理、etc. と日によって変わっていたようで、特に食事に支障はなかった。

セッション

動画リスト

• https://www.youtube.com/user/AmazonWebServices
• https://www.youtube.com/channel/UCdoadna9HFHsxXWhafhNvKw/playlists

Keynotes 気になったスライド

• compute
  • 高性能な instances への対応 ¹
  • fargate for eks、eks インフラの managed
  • networking、言及なし

• cassandra (mcs) の追加、 wide column database

• ai/ml シリーズ

• breaking through barriers ¹
  • outposts (on-prem workload)
  • local zones (low latency to local end-users)
  • wavelength (5G)

Keynotes 気になったサービス

• Amazon CodeGuru
  • Java のみっぽいので色々な言語の対応が待ち遠しい
• Amazon Kendra
  • 企業内ドキュメントの検索は皆困っていて需要があるんだろう
• Amazon Builders' Library
  • 読む必要がある

以下、雑多な参加したセッションの個人メモ

Midnight Madness

Queue up at 8 pm とあったが、開始まで二時間待つのも待機列で一時間ほど掛かるのもどちらも辛い。
ラスベガスにたどり着くまでに体力を消耗しすぎて最後まで参加できず。

ARC411-R - [REPEAT] Reducing blast radius with cell-based architectures

• state machine
• serverless (aws takes care av zone)
• zone based
  • regional control plane -> zone (zone control plane, data plane)
• well architectured framework
• tenets
  • minimize blast radius (spike, etc.)
  • serverless (lambda, kinesis - no zone based)
  • infra as code
• analytics cell router -> analytics cells -> KINESIS -> s3 for datastore -> (engineer, asap, ml, etc.)
• s3 -> asap -> detections cell router -> detection cells -> findings repos
  • router
    • s3 event log -> sns -> (analytics cell router)[sqs -> (lambda -> dynamodb rules)] -> analytics cells
    • rules
      • load balancing, data segregation (source recognized source a to cell a)
      • data replication (prod cells copy to test cells), blue/green deployment
      • never look s3? spof (よくわからんかった）
    • cell
      • apache flink checkpoint and save
• key takeways
  • small vs. large
  • design for failure
  • thinnest possible layer

CON421-R - [REPEAT] Amazon EKS under the hood

• under the hood
  • eks managed vpc | customer managed vpc
    • zone 間はどうふりわけるのか？ -> 53 っぽい
  • eks celluar arch (1 cell = 1 aws account = frontend, cluster events, control plane management, etc.)
  • pipeline -> prow cluster
• enhancement
  • managed node groups (single command, up to date, ha)
    • simplify worker node management, self managed, k8s ecosystem tooling (autoscaler)
  • vision (globally available, easy to use, production ready, cost-effective, high performance)
• snap service mesh on eks
  • eks, envoy, switchboard, spinnaker

ARC335-R - [REPEAT] Designing for failure: Architecting resilient systems on AWS

• RPO, RTO -> mission
• tier 1, 2, ...
• well architected framework: shared responsibility model
• resilient aws infra
  • fault isolation zones: cell-based, multi-az
  • microservice
  • distributed best practices: throttling, retry, circuit breaker
• strategy: backup & restore (multi-region) high level rto, rpo
• strategy: pilot light
• strategy: warm standby
• strategy: active-active
  • (one ex. read locally, write globally)
• s3 - cross-region replication
  • replication time control (new! two weeks ago)
• ebs - snapshot copy
• dynamodb - multi-master, multi-region (no complexity write globally)
• rds:- cross-region read replicas
• inter-region vpc peering -> white paper
• snapchat
  • 99.99% (tier0), 99.95% (tier1), 99% (tier2), 95% (tier3)
  • legacy
    • monolithic, single region -> multi-region active-active
    • repl: dynamodb streams -> stream service -> other regions, etc.
• continuous resilience
  • disaster recovery -> chaos engineering -> continuous resilience

KYN201 - Monday Night Live

目新しい情報はなかったらしい？
席を立つ人が目立った。

KYN202 - Keynote by Andy Jassy

65000+ attendees
3000+ sessions

! transformation

1. senior leadership team conviction and alignment
2. top down aggressive goals
3. train your builders
4. don't let paralysis stop you before you start

97% on premises

• user: goldman saches
• comupute (nitro, chip)
• m6g, r6g, c6g instances (gravition chip)
• inf1 (infrantia chip)
• containers
• fargate for eks https://aws.amazon.com/jp/blogs/aws/amazon-eks-on-aws-fargate-now-generally-available/
  • serverless
• data silos -> data lake
  • s3 access point https://aws.amazon.com/jp/blogs/aws/easily-manage-shared-data-sets-with-amazon-s3-access-points/
  • redshift ra3 instances with managed storage https://aws.amazon.com/jp/blogs/aws/amazon-redshift-update-next-generation-compute-instances-and-managed-analytics-optimized-storage/
  • elasticsearch service (ultrawarm) https://aws.amazon.com/jp/blogs/aws/announcing-ultrawarm-preview-for-amazon-elasticsearch-service/
• database
  • managed cassandra service (wide column) https://aws.amazon.com/jp/blogs/aws/new-amazon-managed-apache-cassandra-service-mcs/
• ml
  • usecase: health care
  • sage maker studio, notebooks, experiments, debugger, model monitor, autopilot
  • fraud detector
  • codeguru (一番盛り上がった）
• contact lens for amazon connect
• kendra
• break on-prem barriers
• outposts GA
  • native aws or vmware cloud on aws
• local zones
• 5G
  • 8 capabilities
  • wavelength

SVS310-R1 - [REPEAT 1] Securing enterprise-grade serverless apps

• speed + security
• security: where to start?
  • identity
    • diy identity store - vulunerable
    • hashed - attask, salt hashed - ...
    • (ok) ssecure remote password protocol (SRP)
    • Amazon Cognito (SRP, etc.)
• x multiple identities o centralize identity management and privilege management
  • delegation: OIDC + OAUTH, federation: SAML, no long-term credentials, rbac
  • delegation
    • aws security token service -> temporary iam creds
  • jwt: identity token, access token, refresh token
  • least privilege, iam condition
• access control to api gateway
  • with cognito
  • allowed -> jwt + context.identity
  • with lambda authorizer -> policy
  • others
    • basic request vailidations on api gateway: parameters, payload with JSON schema
    • CORS
• access control to dynamodb (iam, cognito condition), s3 (iam, bucket policies)
  • s3 access point (multi access control)
• lambda
  • shared responsibility model
  • iam invoke, actions, assume
• common vulnerabilities: ddos
  • aws shield
• common: OWASP top risks to web apps - XSS
  • aws waf filtering rules
• common: SQL injection
  • use prepared
• apply security at all layers
• secure coding practices
  • accessing db creds: x hard coded, ! env or config file, o aws secrets manager, o iam authentication for amazon rds
  • data api for amazon aurora serverless https://docs.aws.amazon.com/ja_jp/AmazonRDS/latest/AuroraUserGuide/data-api.html
  • monolitic functions -> microservies + event-driven architectures (small functions)
  • owasp secure coding practices

STP203 - The modern startup: A look at today’s most successful architectures

• find and eliminate blockers and wait states

  • remove human interaction, self service tools, abstraction layers allow for modular evolution

• enable your developers

  • preparing hackathon

• everything continuous, all the time

  • how long x takes?

• keep your systems simple

  • gall's law

• democratize your data

• develop a culture of openness

• legacy: ecs containers

  • goal: 100x clients, faster, more data

• shift

  • docker -> serverless, tdd, devops, devs, and QA are all one

CON323-R1 - [REPEAT 1] AWS App Mesh under the hood

• working backward
  • customer want: routing, observability, security, transparency
  • arch
    - proxy
    - virtual node, virtual service, virtual router
• design
  • frontend service, envoy management service, transformer
  • frontend service
    • authentication
    • authorization
    • validation
    • persistence
      • picking
        • customer configuration/metadata, relationships between resources, serializable cross-key/table -> JournalDB
  • transformer
    • mesh id -> buckets (split brain -> allow extra)
    • event-driven processing
      • reactive, efficient, creates backlog, requires reconciliation
      • level-driven processing: bounded workload (bimodal behavior), self-healing, less efficient
  • envoy
    • envoy data store
      • synthesized envoy configuration, relationships node to manifest, eventually consistency, single key -> dynamodb
    • discovery - listeners - routes - secrets - clusters - endpoints
    • management
      • actor system(connection manager, manifest manager, aws cloud map, aws certificate manager)
• operations (canary)
  • deployment: first (container images and cloudformation templates), second, remaining
  • monitoring: rate, errors, duration (+ cloudwatch log insight)

AIM207-R5 - [REPEAT 5] Get started with AWS DeepRacer

CON325-R - [NEW LAUNCH!] [REPEAT] Enabling application-first thinking with Amazon ECS capacity providers

• orchestraction ecs/eks, compute ec2/fargate
• containers: 150%+ growth, 80% share (cloud)
• ecs internal customers (sagemaker, etc) - dogfooding
• infrastructure first
  • terminology: ecs cluster (namespace), ecs task (like pods), service (many tasks)
  • start ec2 instances first, and then run ecs tasks (infra first)
  • task placement: available instances -> resources (like resource limit) -> placement constraints (like affinity) -> strategy
  • infra first: available instances not found, spread placement strategies, scaling (cloudwatch metrics -> alarm auto scaling groups: metrics are based only on existing tasks and resources)
• application first
  • tenets: applications own their requirements, infrastructure responds to application requirements
• ECS Capacity Providers
  • ecs cluster ( ecs cp ( ec2 asg (ec2 instances)))
  • ecs cp
    • abstracts capacity
    • 0-10 /cluster
    • used by tasks/services
  • run tasks under cp without instances -> instances start -> tasks placed
  • ec2 spot / fargate spot
    • price up to 90%/80%
    • reclaimed by ec2/ reclaimed
    • instances pools/automatic

ARC349-R1 - [REPEAT 1] Beyond five 9s: Lessons from our highest available data planes

• api caller -> | (api) control plane - config -> data plane (server) | <- client
• 10 patterns
  • insist on the highest standards
  • raise the bar for testing
    • 1000s unit tests, 100s integration tests, pre-prod env, roll-forward/back testing
  • be technically fearless
    • mitigate fear with professionalism, testing, and open-@minded scrutiny
    • aws lindbergh award
  • modeling
  • focus on blast radius
    • shuffle sharding https://aws.amazon.com/jp/blogs/architecture/shuffle-sharding-massive-and-magical-fault-isolation/
  • modular separation
  • double down on simplicity
    • saying no to many features,
  • unreasonable redundancy
    • striping, shuffle sharding
    • teeny cache
  • static stability
  • degrade gracefully

KYN204 - Keynote by Dr. Werner Vogels

DOP210-L - Leadership Session: Developer Tools on AWS

• external/internal, builder.tools
• dev/test
  • delivery: dev/test -> review -> pre-production -> production
  • dev/test
    • use cloud desktops at amazon
    • use cloud9 internally
    • support aws toolkit for third party IDE
      • debugging timeline 30min -> 10 min
      • demo
• review
  • codecommit supports approval rules
  • codeguru review/profiler
• ci/cd
  • 2001: monolith -> 2002: 2-pizza teams (devops, full ownership, full accountability, focused innovation)
    • delivery pipelines (dev tool team)
  • delivery: source -> build -> alpha (pre-production, automated tests) -> beta (pre-production, automated integration tests, load/perf tests, browser tests) -> gamma (pre-production, automated integration tests, synthetics tests, api smoke tests) -> one az/fractional (production, synthetic monitoring) ... -> one region...
  • delivery is pessimistic
  • pipeline blockers: time windows, pipeline policies, ocverage, review, security scans, dependency updates, etc.
• modern applications
  • monolithic -> n-tier -> micoservices & serverless
  • aws cdk
• a look ahead
  • security

SEC209-R1 - [REPEAT 1] Getting started with AWS identity

• before the cloud (firewall, etc.) -> in the cloud (iam)
• iam roles: recommendation - have at least admin and readonly
• authentication and authorization
  • iam role -> resource
    • short-term creds for iam role
    • if any policy denies -> access denied
    • if some policy allows -> allow
    • otherwise denied
• cross account: recommendation - keep it simple
  • resource based policy
  • role trust policy

1. on-prem が 97% 残ってるのでそれを取っていく感じの進化？ ↩ ↩²