zak

11 ms·

1.
by zak 3y ago
For future reference, the team looked into this, and it appears that the interruptions you experienced were specific to your project and a small number of other projects. The vast majority of TRC projects should see much longer Cloud TPU up
2.
by zak 3y ago
Could you share a few technical details about the issues you've encountered with TF / JAX / PyTorch on Cloud TPUs? The overall Cloud TPU user experience improved a whole lot when we enabled direct access to TPU VMs, and I bel
3.
by zak 3y ago
As mentioned in another comment, it sounds like you're using preemptible TRC TPU quota. If you use on-demand TRC TPU quota instead, that should improve your uptime substantially.
4.
by zak 3y ago
It sounds like you're primarily using preemptible TPU quota, which doesn't come with any availability or uptime expectations at all. By default, the TRC program grants both on-demand quota and preemptible quota. If you are able to
5.
by zak 3y ago
A few quick comments: > But it’s important for hobbyists and tinkerers to be able to participate in the AI ecosystem Totally agree! This was a big part of my original motivation for creating the TPU Research Cloud program. People sometim
6.
by zak 3y ago
Actually, the TPU Research Cloud program is still going strong! We've expanded the compute pool significantly to include Cloud TPU v4 Pod slices, and larger projects still use hundreds of chips at a time. (TRC capacity has not been rec
7.
by zak 4y ago
No, Cloud TPUs support JAX, PyTorch, and TensorFlow, and the new TPU VM architecture provides enough low-level access that users could add support for additional frameworks themselves if they are willing to put in substantial effort.
8.
by zak 4y ago
If you haven't used Cloud TPUs in a while, I'd encourage you to try them now with TPU VMs and the latest versions of JAX, PyTorch / XLA, or TensorFlow. We've gotten a lot of positive feedback from customers and TRC users
9.
by zak 4y ago
We try to make it easy to switch back and forth between Cloud TPUs and other hardware platforms using JAX, PyTorch, and TensorFlow. This is a difficult technical challenge, but the XLA compiler helps a lot, and switching is easier now than
10.
by zak 4y ago
Thanks very much! We've come a long way, but there is always more interesting work required to keep up with the deep learning frontier and enable Cloud TPU customers and TRC users to expand it further.
11.
by zak 4y ago
I started the TRC program alongside the Cloud TPU program to make interesting amounts of ML compute available to a broad group of creative people, not only to academic researchers. The TRC program welcomes hobbyists, artists, students, inde
12.
by zak 4y ago
Yes, there are a couple of ways to use Cloud TPUs at lower priority and lower cost. If you are a hobbyist, I highly recommend trying out Cloud TPUs for free via the TPU Research Cloud: https://sites.research.google/trc/
13.
by zak 4y ago
No, Vectorflow is not supported out of the box, and I'm not sure the workloads it targets are the right fit for Cloud TPU hardware. However, be sure to check out the "Ranking and recommendation" section of the linked blog pos
14.
by zak 4y ago
The MLPerf 1.0 results provided an apples-to-apples comparison of large-scale TPU and GPU systems across several ML workloads: https://cloud.google.com/blog/products/ai-machine-learning/g... In MLPerf 1.1, we
15.
by zak 4y ago
Thanks, Frank! You personally helped more Cloud TPU and TRC users than I can count, and you always came through something needed to get done and fast. I really appreciated it!
16.
by zak 4y ago
I started pitching the Cloud TPU program in 2016. Many, many people have contributed since then to build the products that are available today. Google is a large and complicated place, but we're getting closer to providing the magical
17.
by zak 4y ago
Thanks, and congratulations to many others across many teams who have supported the Cloud TPU program over the years!
18.
by zak 4y ago
We love Colab and would love to upgrade the Colab TPU integration to support TPU VMs! No timeframe yet, but the right folks across JAX / Colab / Cloud TPU are very aware of this issue.
19.
by zak 4y ago
In the previous Cloud TPU architecture, PyTorch and JAX users had to create a separate CPU VM for every remote TPU host and arrange for these CPU hosts to communicate indirectly with the TPU hosts via gRPC. This was cumbersome and made debu
20.
by zak 4y ago
Founder of the Cloud TPU program here. If you'd like to experiment with TPU VMs for free and are willing to share your work with the world somehow (e.g. via publications or open-source projects), you can apply to participate in the TPU
21.
by zak 4y ago
Here's a quick overview of TPUs from last year's Google I/O keynote: https://www.youtube.com/watch?v=XFFrahd05OM&t=1565s
22.
by zak 4y ago
Yes, TPU VMs dramatically improve the Cloud TPU user experience. You now have direct access to the VM on each TPU host whether you are using JAX, PyTorch, or TensorFlow, which provides a lot more flexibility and control and can often improv
23.
by zak 4y ago
Thanks for the very kind feedback! We've wanted to provide TPU VMs since the beginning of the Cloud TPU program, and I'm delighted that you're enjoying them. Many people across Google contributed to this launch. We're de
24.
by zak 7y ago
We only see code when customers open-source it or otherwise explicitly share it with us. We are directly in touch with several customers who are using the PyTorch / TPU integration, so we hear feedback from them, and we also run a vari
25.
by zak 7y ago
(I'm one of the Cloud TPU product leads) We've seen multiple BERT-related PyTorch models training successfully on Cloud TPUs, including training at scale on large, distributed Cloud TPU Pod slices. Would you consider filing a GitH
26.
by zak 7y ago
When comparing different hardware configurations within or between public clouds, power measurements generally aren't available, whereas prices or realistic price estimates generally are.
27.
by zak 7y ago
Sure. The $2.48/hour per V100 GPU on GCP does not include the price of the CPU host; that is purely the price to rent a single accelerator. By contrast, a network-attached Cloud TPU v3 device includes both a CPU host and four connected
28.
by zak 7y ago
Yes, Cloud TPU Pods are designed to train much larger models on much larger datasets. And, as you mention, if you are willing to adjust your model architectures and training algorithms to take full advantage of the hardware, you can sometim
29.
by zak 7y ago
Author of the blog post here. As mentioned in other comments, I'd recommend doing a performance-per-dollar comparison in addition to looking at this pure performance comparison at maximum scale.
30.
by zak 7y ago
Author of the blog post here. I'd recommend doing a performance-per-dollar comparison before drawing this conclusion.

More ›