BigQuery Omni and S3
Transcript
we gave an example of Athena a few months ago where you could take your S3 buckets and then perform SQL queries against that bucket using aws's Athena service really interesting kind of how do I replace Splunk with a native AWS service without the cost of managing these block infrastructure one of the disadvantages to this approach is that Athena is relatively slow especially compared to something like a Splunk so what if you needed more real-time analysis of that data and out of AWS are other Solutions such
as redshift but there's actually something that sits inside of AWS that is kind of low-key which is Google big Curie Omni it is the service operated with by Google inside of aws's cloud bigquery can also analyze data stored in your S3 buckets hmm that sounds like something that would be expensive because we've heard about these transient costs with the cloud providers if I'm accessing S3 directly from gcp then I have to pay transient Network fees from my S3 endpoint into my gcp bigquery compartment but this
compute is directly in S3 so if I'm accessing the data within the same region I don't have to pay the transfer rates it's operated by Google and I get the performance benefits of the bigquery infrastructure what's the catch well it's cost we're not getting something for nothing gcp has to make some type of profit on top of the compute they're using within AWS to make this managed service work so we get kind of this hidden cost with AWS Athena I'm paying five dollars per five terabytes
analyze which sounds the same as the five dollars per terabyte that I'm getting charged by big Curie the difference is when I get charged that data rate AWS Athena charges me the five dollars per compressed data five terabytes of compressed data gcp charges me five dollars uncompressed so gcp is seeing the cost Savings of using compression where and not passing that on to the customer whereas AWS is passing that cost savings on
