How did we reduce the cost of the data storage solution by 95%?
Implementation
GCP
Technology
BigQuery, Cloud Storage
Tooling
Golang
Team
2 Engineers
Scale
0.5 PB of data
BENEFITS
95% cost reduction
Summary:
For audit purposes, we were required to keep certain logs of in-game transactions for a couple of years and make them available for occasional querying. To reduce complexity and costs, we were able to unify the systems for gathering these logs and decrease the overall solution cost by 95% during the design process.
Challenges:
Data volume of over 0.5PB
Multiple version of the system
Data stored on remote sites (due to legal requirements)
Solution:
Analysie the data
Redesign data format to decrease overhead
Gather, Classify and structure the data
Prepare cost efficient data-lake solution powerd by BigQuery, Cloud Storage and clever data placement