Cloud TPU pricing

TPU is short for tensor processing unit. A TPU processes tensors which are fancy matrices (MORE ON THIS) which is handy when you have a machine learning problem that you want to train or test. However, Google Cloud TPUs can get pretty expensive, and they require having a Compute Engine Virtual Machine (VM) in order to connect and use them. We are going to break down the costs involved with having and using Google Cloud TPUs and their VMs.

TPUs and VMs are billed even if you are not actively using them. Basically, as long as you have a TPU node you are being charged no matter if you use it or not. There are two ways of charging TPUs. One is called single device pricing (that means you only have one TPU node running and that one node can not be connected to others) and the other is called TPU pod type pricing (for clusters containing multiple TPU devices, accessible only to those who have a 1 or 3 year commitment).

Single device TPU types are billed in one-second increments and are available at either an on-demand or PREEMPTIBLE price. Preemptible means your TPU can shut down at any time if the overall usage is high, which results in a lower price, but also means you have to take those unexpected shutdowns into account when programming your applications. The table below shows the pricing for europe-west4 Netherlands region (we used version 3-8).

Charges of TPUs and VMs are separate so these prices you see are only for TPUs. You can learn more about VM pricing HERE. It’s pretty clear that this kind of solution for machine learning problems can get pretty expensive, fairly quick, which we will look at a little bit later.

VMs vs Containers vs Serverless

Cloud Functions and Cloud TPU (along with its Compute Engine VM) represent two very different approaches to cloud computing in terms of how much control over the resources you have, and how much you pay for it.

Virtual machines (VMs) are part of Infrastructure as a Service (IaaS). It’s a model which gives you the most control over your resources, but with control also comes responsibility for maintaining and securing the OS running on your VM. VMs also require additional billable resources such as persistent storage and networking.

The basic building blocks of Platform as a Service (PaaS) are containers. Your code along with all of its dependencies is packaged into a container which can be easily run on different operating systems and environments. However, the real power of containers manifests itself in the form of orchestration, which allows you to automatically deploy and scale your application. You can learn more about Kubernetes, the most prominent orchestration software for Docker containers, in ENA’S BLOG POST.

Functions as a Service (FaaS) represent the highest level of abstraction in cloud computing where you can still program the application. FaaS are also called serverless functions, not because they don’t run on a server, but because that server layer is completely hidden from the user and managed by the cloud provider. Resources and execution time are typically much more limited here, for example you can’t have a GPU connected to a Cloud Function on GCP, and one instance can only be run for 9 minutes before it times out.

Just briefly, Software as a Service (SaaS) represents user applications which are billed with a pay-as-you-go model. You cannot make changes to such applications. Examples include Office 365, Salesforce, Slack, and many others.

 1.1 On PREMISE VS. SERVERLESS – Customers love solutions (customers-love-solutions.com)

So, which to choose? Well, it depends on what your application is. As people are moving to a component based approach away from big monolithic applications, VMs are largely losing popularity to container based solutions, because of their increased flexibility. Maybe your use case requires a simple enough solution which can be run as a serverless function. Don’t take simple too strongly here, as we’ve proved you can do some pretty complex computation such as training neural networks in serverless functions.

Other costs (Pub/Sub, Storage, SQL)

PUB/SUB PRICING has three components: throughput costs for message publishing and delivery, storage fees when retaining all or acknowledged messages or creating snapshots, and egress costs when crossing a zonal or regional boundary. It is important to mention that storage of unacknowledged messages is free. Google also offers a cheaper PUB/SUB LITE option which can be as much as an order of magnitude less expensive, but offers less features and requires more manual configuration.

In our specific use case we haven’t configured storage for acknowledged messages, as that is taken care of by Persistor because all of our resources were in the same region, no egress fees were incurred. This means we would have paid only for the throughput (message publishing & delivery), however the first 10 GiB in a calendar month are free, so basically Pub/Sub didn’t cost us anything!

Basic CLOUD STORAGE usage is billed according to data storage, network usage, operations usage, and others if using some additional advanced features. Data storage is priced depending on the continent and is measured in gigabytes per month. Network usage pricing is split into egress (data sent from GCS) and ingress (data sent to GCS). Operations are actions that make changes to, or retrieve information about buckets and objects in Cloud Storage. They are separated into classes, and billed in units of 10,000. As part of Always Free usage limits, 5 GB/month, 5,000 Class A Operations, and 50,000 Class B Operations, and 1 GB of network egress are free, but only in three United States regions.

CLOUD SQL PRICING depends on whether you’re using a dedicated or shared CPU. Dedicated instances are charged by the number of cores and memory amount. Shared-core instances are charged for every second that the instance is running. It is important to mention that shared-core instances are not covered by the CLOUD SQL SERVICE LEVEL AGREEMENT (SLA) and are not recommended for production environments. All Cloud SQL instances incur storage and networking costs, with ingress to Cloud SQL being free.

Experiment & Cost Comparison

Let’s compare using two totally different cloud solutions – Cloud Functions and Cloud TPU – for solving the same machine learning problem – training a convolutional neural network (CNN) on the MNIST dataset. The training set consists of 60,000 samples of 28×28 monochrome images. When using Cloud Functions, the data was published 6 times, separated into 10 messages with 1,000 samples each. Keep in mind that Persistor is also two Cloud Functions which could also potentially incur costs (hence the high number of invocations). The price for all of this? Eight measly cents ($0.08).

Looking at Cloud TPU, it seems significantly more expensive, and it is, but maybe not as much as this image would first suggest. Cloud TPU incurs costs when it’s in the READY state, so not necessarily only when performing computations. We were following Google’s RUNNING MNIST ON CLOUD TPU tutorial and it took us some time to set everything up correctly. During all of that, Cloud TPU was charged even though it wasn’t doing any neural network training or other kind of computation.

All in all, it takes the TPU less than 5 minutes to train a neural network on the whole MNIST dataset. Even when we look at the price for such a short interval (approx. $0.72), it’s still an order of magnitude higher compared to using Cloud Functions. You also should take into account that Cloud TPU needs an underlying Compute Engine VM to run, which can incur significant costs related to CPU, memory, storage, and networking usage, adding unnecessary overhead costs.