【GCP】Professional Cloud Architect - Case Study

MountKirk Games

Background

• a new game expect to be very polular.
• Plan to deploy backend to Compute Engine.
• Capture streaming metric
• Run intensive analytics
  • Cloud Data Loss Prevention API
• integrate with a managed NoSQL database
  • Migrate MySQL to BigTable or Datastore

Business requirements

• Increase to a global footprint
  • Use a global HTTP load balancer that serves managed instances groups that have autoscaling enabled
• Improve uptime-downtime
• Increase efficiency of the cloud resources
  • Migrate MySQL to BigTable or Datastore
• Reduce latency to all customers

Technical requirements

• Backend platform
  • Dynamically scale up or down based on game activity
    • Managed Instance Group
    • Use Kubernetes Engine if they want create microservices.
  • Connect to a transactional database service to manage user profiles and game state
    • Cloud Spanner
  • Store game activity in a timeseries database service for future analysis
    • BigTable for store datas
    • BigQuery for analysis
  • No data lost due to processing backlogs
    • Cloud Pub/Sub
  • Custom Linux distro
    • Cannot use App Engine. (Standard or flexable)
    • Compute Engine Only
    • Use a global HTTP load balancer that serves managed instances groups that have autoscaling enabled
• Analytics platform
  • Realtime Analytics
    • Streaming Pipeline: Cloud Pub/Sub -> Cloud Dataflow -> BigQuery
    • Batch Pipeline: Cloud Storage -> Cloud Dataflow -> BigQuery
  • Dynamically scale up or down based on game activity
    • Use BigQuery for analytics
  • Process incoming data on the fly directly from the game servers
    • Use Cloud Pub/Sub handles streaming data in real time
  • Process data that arrives late because of slow mobile networks
  • Allow queries to access at least 10 TB of historical data
    • Big Query
  • Process files uploaded by user's mobile

Advices

• Using Stackdriver Logging and Monitoring
• Use a lighter base image such as Alpine Linux improve deployment times

TerramEarth

Background

• 20 million vehicles that collect 120 fields for data per second
• data is stored locally
• data is downloaded via a maintenanace port.
• 200,000 vehicles are connected to a cellular network, allowing to collect data directly.

Existing technical environment

• Linux and Windows-based systems that reside in a single U.S. west-coast-based data center. - - gzip CSV files from the field and upload via FTP
• Place the data in their data warehouse.
• Aggregated reports are based on data that is three weeks old.

Business requirements

• Decrease unplanned vehicle downtime to less than one week
  • Use Streaming Pipeline
    • Cellular -> Cloud Pub/Sub -> Cloud Storage -> Cloud Dataflow -> BigQuery
  • Use Batch Pipeline
    • User -> Cloud Storage -> Cloud Dataflow -> BigQuery
• Support the dealer network with more data on how their customers use their equipment to better position new products and services
  • Use Google Data Studio to create live charts that read directly from BigQuery. Give dealer representatives view rights to these charts to gain better understanding.
• Have the ability to partner with different companies-especially with seed and fertilizer suppliers in the fast-growing agricultural business-to create compelling joint offerings for their customers

Technical requirements

• Decrease latency to the American Midwest and East Coast
  • Upload data to regional bucket of location
• Create a backup strategy
• Increase security of data transfer from equipment to the data center
• Improve data in the data warehouse
  • Use a BigQuery with table partitioning
• Use customer and equipment data to anticipate customer needs

Applications

• a custom python application reads uploaded files, then writes to the data warehouse.
• a daily report to see what equipment needs repair.

Advices

• Use Cloud Pub/Sub handles streaming data in real time
• Use Cloud Data Studio to replace current analytics software
• Use Cloud Endpoints manage and project their APIs

Dress4Win

Background

• web app and mobile application
• committing to a full migration to a public cloud
  • MySQL： Set up a MySQL replica/slave in Google Cloud using Cloud SQL and configure it for asynchronous replication from the MySQL master server on-premises until cutover.
  • Rabbit MQ： Cloud Pub/Sub
  • NAS： Cloud Storage bucket
  • SAN： Persistent disk
  • Apache Hadoop/Spark： Dataproc
• moving their development and test environments
• building a disaster recovery site

Technical environment

• Databases:
  • MySQL. One server for user data, inventory, static data
    • Cloud SQL migration service
    • MySQL 5.7
    • 8 core CPUs
    • 128 GB of RAM
    • 2x 5TB HDD (RAID 1)
• Compute
  • 40 web application servers providing micro-services based APIs and static content
    • Managed Instance group
    • Tomcat - Java
    • Nginx
    • Four core CPUs
    • 32 GB of RAM
  • 20 Apache Hadoop/Spark servers:
    • Migrate jobs to Dataproc
    • Data analysis
    • Real-time trending calculations
    • Eight core CPUs
    • 128 GB of RAM
    • 4x 5 TB HDD (RAID 1)
  • Three RabbitMQ servers for messaging, social notifications, and events:
    • Migrate to Cloud Pub/Sub
    • Eight core CPUs
    • 32GB of RAM
  • Miscellaneous servers:
    • Compute Engine
    • Jenkins, monitoring, bastion hosts, security scanners
    • Eight core CPUs
    • 32GB of RAM
• Storage appliances:
  • iSCSI for VM hosts
  • Fibre channel SAN - MySQL databases
  • 1 PB total storage; 400 TB available
  • NAS - image storage, logs, backups
  • 100 TB total storage; 35 TB available

Business requirements

• Build a reliable and reproducible environment with scaled parity of production
• identity and access management (IAM) best practices for cloud to improve security
• Improve business agility and speed of innovation through rapid provisioning of new resources Analyze and optimize architecture for performance

Technical requirements

• Easily create non-production environments
  • Deployment Manager
• Implement an automation framework for provisioning resources
  • Deployment Manager
• Implement a continuous deployment process for deploying applications
  • pre-built Jenkins image form MarketPlace
• Support failover of the production environment to cloud during an emergency
• Encrypt data on the wire and at rest
• Support multiple private connections between the production data center and cloud environment.
  • Dedicated Interconnect or Partner Interconnect