Cloud Hosting

Digital Realty's CEO and more on what generative AI means for the data center industry

A potential shift in the nature of workloads will filter down to the wider data center industry, impacting how they are built and where they are located.

Digital Realtys CEO Andy Power believes that generative AI will lead to a monumental wave of demand.

It's still new as to how it plays out in the data center industry, but it's definitely going to be large-scale demand. Just do the math on these quotes of spend and A100 chips and think about the gigawatts of power required for them.

When he joined the business nearly eight years ago we were moving from one to three megawatt IT suites, and we quickly went to six to eight, then tens, he recalled. I think the biggest building we built was 100MW over several years. And the biggest deals we'd sign were 50MW-type things. Now you're hearing some more deals in the hundreds of megawatts, and I've had preliminary conversations in the last handful of months where customers are saying talk to me about a gigawatt.

For training AI models, Power believes that well see a change from the traditional cloud approach which focuses on splitting up workloads across multiple regions while keeping it close to the end user.

Given the intensity of compute, you cant just break these up and patchwork them across many geographies or cities, he said. At the same time, you're not going to put this out in the middle of nowhere, because of the infrastructure and the data exchange.

These facilities will still need close proximity to other data centers with more traditional data and workloads, but the proximity and how close that AI workload needs to sit relative to cloud and data is still an unknown.

He believes that it will still be very major metro focused, which will prove a challenge because youre going to need large swaths of contiguous land and power, but its harder and harder to find a contiguous gigawatt of power, he said, pointing to transmission challenges in Virginia and elsewhere.

As for the data centers themselves, plain and simple, it's gonna be a hotter environment, you're just going to put a lot more power-dense servers in and you're gonna need to innovate your existing footprints, and your design for new footprints, he said.

We've been innovating for our enterprise customers in terms of looking at liquid cooling. It's been quite niche and trial, to be honest with you, he said. We've also been doing co-design with our hyperscale customers, but those have been exceptions, not the norms. I think you're gonna see a preponderance of more norms.

Moving forward, he believes that you'll have two buildings that will be right next to each other and one will be supporting hybrid cloud. And then you have another one next to it that is double or triple the size, with a different design, and a different cooling infrastructure, and a different power density.

Amazon agrees that large AI models will need specialized facilities. Training needs to be clustered, and you need to have really, really large and deep pools of a particular capacity, AWS Chetan Kapoor said.

The strategy that we have been executing over the last few years, and we're going to double down on, is that we're going to pick a few data centers that are tied to our main regions, like Northern Virginia (US-East-1) or Oregon (US-West-2) as an example, and build really large clusters with dedicated data centers. Not just with the raw compute, but also couple it with storage racks to actually support high-speed file systems.

On the training side, the company will have specialized cluster deployments. And you can imagine that we're going to rinse and repeat across GPUs and Trainium, Kapoor said. So there'll be dedicated data centers for H100 GPUs. And there'll be dedicated data centers for Trainium.

Things will be different on the inference side, where it will be closer to the traditional cloud model. The requests that we're seeing is that customers need multiple availability zones, they need support in multiple regions. That's where some of our core capability around scale and infrastructure for AWS really shines. A lot of these applications tend to be real-time in nature, so having the compute as close as possible to the user becomes super, super important.

However, the company does not plan to follow the same dense server rack approach of its cloud competitors.

Instead of packing in a lot of compute into a single rack, what we're trying to do is to build infrastructure that is scalable and deployable across multiple regions, and is as power-efficient as possible, Kapoor said. If you're trying to densely pack a lot of these servers, the cost is going to go up, because you'll have to come up with really expensive solutions to actually cool it.

Googles Vahdat agreed that we will see specific clusters for large-scale training, but noted that over the longer term it may not be as segmented. The interesting question here is, what happens in a world where you're going to want to incrementally refine your models? I think that the line between training and serving will become somewhat more blurred than the way we do things right now.

Comparing it to the early days of the Internet, where search indexing was handled by a few high-compute centers but is now spread across the world, he noted: We blurred the line between training and serving. You're gonna see some of that moving forward with this.

While this new wave of workload risks leaving some businesses in its wake, Digital Realtys CEO sees this moment as a rising tide to raise all ships, coming as a third wave when the second and first still haven't really reached the shore.

The first two waves were customers moving from on-prem to colocation, and then to cloud services delivered from hyperscale wholesale deployments.

Thats great news for the industry, but one that comes after years of the sector struggling to keep up. Demand keeps out-running supply, [the industry] is bending over coughing at its knees because it's out of gas, Power said. The third wave of demand is not coming at a time that is fortuitous for it to be easy streets for growth.

Our largest feature ever looks at the next wave of computing

17 Apr 2023

For all its hopes of solving or transcending the challenges of today, the growth of generative AI will be held back by the wider difficulties that have plagued the data center market - the problems of scale.

How can data center operators rapidly build out capacity at a faster and larger scale, consuming more power, land, and potentially water - ideally all while using renewable resources and not causing emissions to balloon?

Power constraints in Northern Virginia, environmental concerns, moratoriums, nimbyism, supply chain problems, worker talent shortages, and so on, Power listed the external problems.

And that ignores the stuff that goes into the data centers that the customer owns and operates. A lot of these things are long lead times, with GPUs currently hard for even hyperscalers to acquire, causing rationing.

The economy has been running hot for many years now, Power said, And it's gonna take a while to replenish a lot of this infrastructure, bringing transmission lines into different areas. And it is a massive interwoven, governmental, local community effort.

While AI researchers and chip designers face the scale challenges of parameter counts and memory allocation, data center builders and operators will have to overcome their own scaling bottlenecks to meet the demands of generative AI.

We'll continue to see bigger milestones that will require us to have compute not become the deterrent for AI progress and more of an accelerant for it, Microsofts Nidhi Chappell said. Even just looking at the roadmap that I am working on right now, it's amazing, the scale is unprecedented. And it's completely required.

As we plan for the future, and try to extrapolate what AI means for the data center industry and humanity more broadly, it is important to take a step back from the breathless coverage that potentially transformational technologies can engender.

After the silicon boom, the birth of the Internet, the smartphone and app revolution, and cloud proliferation, innovation has plateaued. Silicon has gotten more powerful, but at slower and slower rates. Internet businesses have matured, and solidified around a few giant corporations. Apps have winnowed to a few major destinations, rarely displaced by newcomers. Each new smartphone generation is barely distinguishable from the last.

But those who have benefitted from the previous booms remain paranoid about what could come next and displace them. Those who missed out are equally seeking the next opportunity. Both look to the past and the wealth generated by inflection points as proof that the next wave will follow the same path. This has led to a culture of multiple false starts and overpromises.

The metaverse was meant to be the next wave of the Internet. Instead, it just tanked Meta's share price. Cryptocurrency was meant to overhaul financial systems. Instead, it burned the planet, and solidified wealth in the hands of a few. NFTs were set to revolutionize art, but rapidly became a joke. After years of promotion, commercial quantum computers remain as intangible as Schrodingers cat.

Generative AI appears to be different. The pace of advancement and the end results are clearly evidence that there are more tangible use cases. But it is notable that crypto enthusiasts have rebranded as AI proponents, and metaverse businesses have pivoted to generative ones. Many of the people promoting the next big thing could be pushing the next big fad.

The speed at which a technology advances is a combination of four factors: The intellectual power we bring to bear, the tools we can use, luck, and the willingness to fund and support it.

We have spoken to some of the minds exploring and expanding this space, and discussed some of the technologies that will power what comes next - from chip-scale up to data centers and the cloud.

But we have not touched on the other two variables.

Luck, by its nature, cannot be captured until it has passed. Business models, on the other hand, are usually among the easier subjects to interrogate. Not so in this case, as the technology and hype outpace attempts to build sustainable businesses.

Again, we have seen this before with the dotcom bubble and every other tech boom. Much of it is baked into the Silicon Valley mindset, betting huge sums on each new tech without a clear monetization strategy, hoping that the scale of transformation will eventually lead

to unfathomable wealth.

Higher interest rates, a number of high-profile failures, and the collapse of Silicon Valley Bank has put such a mentality under strain.

At the moment, generative AI companies are raising huge sums on the back of wild promises of future wealth. The pace of evolution will depend on how many can escape the gravity well of scaling and operational costs, to build realistic and sustainable businesses before the purse strings inevitably tighten.

And those eventual winners will be the ones to define the eventual shape of AI.

We do not yet know how expensive it will be to train larger models, nor if we have enough data to support them. We do not know how much they will cost to run, and how many business models will be able to bring in enough revenue to cover that cost.

We do not know whether large language model hallucinations can be eliminated, or whether the uncanny valley of knowledge, where AIs produce convincing versions of realities that do not exist, will remain a limiting factor.

We do not know in what direction the models will grow. All we know is that the process of growth and exploration will be nourished by ever more data and more compute.

And that will require a new wave of data centers, ready to meet the challenge.

13 Jul 2023

13 Jul 2023

13 Jul 2023

13 Jul 2023

13 Jul 2023

13 Jul 2023

Go here to see the original:
Generative AI & the future of data centers: Part VII - The Data Centers - DatacenterDynamics