Cloud Hosting

By Allison Proffitt

May 12, 2022 | At Bio-IT Worlds 20th anniversary event last week, Chris Dagdigian and friends from BioTeam, once again closed out programming with a rapid-fire look at the bio-IT landscape and an IT trends assessmentcalling out what is working, whats not, and how its all evolving.

Repeating a format he introduced last fall, Dagdigian led a panel of speakers, each commenting on their own experience with bio-IT trends. This year all four speakers were experienced BioTeam consultants, but Dagdigian also flagged Twitter friends who can speak freely including James Cuff (@DrCuff, hear his Trends from the Trenches podcast here), Chris Dwan (@fdmnts), @hpcguru, and Corey Quinn (@QuinnyPig).

From his personal vantage pointhaving given an IT trends talk at Bio-IT World since 2009Dagdigian began by outlining the trends that have held firm over the past decade. He still starts every year, Dagdigian said, with the existential dread. Science has always changed more rapidly than IT can keep up, and most certainly faster than your IT budget renews. This remains a problem, Dagdigian said, and there is a real risk when IT builds a wrong solution for the scientist.

Cloud, he repeated, remains a capability play, not a cost savings strategy. Capability and flexibility still justify cloud adoption, they do not, however, justify a multi-cloud approach. A multi-cloud strategy is definitely dumb, Dagdigian said, while a hybrid cloud approach is absolutely fine. Multi-cloud requires developers to devolve applications to the lowest common API denominator. Its a degraded experience, he said, unless you were all in on Kubernetes, which can reasonably port between AWS, Google Cloud, and Microsoft Azure. In his trademark bluntness, Dagdigian said any company with a multi-cloud strategy is a red flag for poor senior leadership.

As in years past, moving and managing data is a pain, Dagdigian said, and he again threatened to call out scientists who build careers and publications lists on data intensive science but refuse to take responsibility for their own data.

Its a shared responsibility model. My job as an IT person is to provide you with safe, durable storage options that are fit for purpose and aligned with what youre trying to do. The combo between science and IT is to provide end users with tools to manage, govern, pull actionable insights, understand what were actually storing. But finally end users have to take some responsibility. Thats the sort of missing piece of the equation. It is wildly inappropriate for IT to make a lot of storage and data management decisions, he said.

Dagdigian deemed many of the problems that weve struggled with in years past solved problems including compute, networking, and storage. He called compute mostly a financial planning exercise and flagged Internet2 and PetaGene as solid networking options that are no longer hard, risky, or exotic.

He pointed to many vendors in the Bio-IT space that can help with storage that all have strong track records and referenceable customers. He advised starting with object storage or scale-out NASonly exploring something else if business or scientific needs require.

So Smug, So Wrong

But one of the great attractions to Dagdigians annual insights is his willingnesseven delightin point out his past errors. He flagged his own storage failed prediction with zeal: The future of scientific data at rest is object storage, he recounted on a slide, and attributed the quote to some jerk.

It sounded good! Object storage can be deployed on premises and in the cloud. Metadata tagging is fantastic for scientific data management and search. And object storage is purpose built for a FAIR future in which humans are not the dominant data consumers.

I am completely, utterly, totally wrong on this, he said. Were still using POSIX Windows or Linux flavored storage.

It turns out, Dagdigian conceded, scientists do still assume that humans are doing most of the folder browsing, and neither commercial code nor open-source code is object-aware. Scientists who just need to transform data in R or Python dont have the bandwidth to learn object storage.

In fact, he flagged a death of tiered storage. Machine learning and AI have messed up long-held storage design patterns in the past three to four years, he said.

The concept of having an archive tier or a nearline tier or a slow tier doesnt make a lot of sense. If youre talking about machine learning or AI, youre really churning through all of your dataold and newall the time. Youre constantly reevaluating, retraining, pulling out different training sets, Dagidian said. I no longer can get away with tiers of different speed and capacity. If I need to satisfy the ML and AI people, you pretty much need one single tier of performant storage.

The vendor landscape on this new storage structure is crowded, he said, but he highlighted VAST Data, Weka, Hammerspace, and DellEMC.

COVID-Era Trends

Next, Dagdigian turned his attention to the trends arising the past year or two, starting with one of his single biggest obnoxious problems: the scarcity of GPUs on the cloud, particularly in Amazons US-East-1 availability code. One Boston-based client is building their first AWS footprint in US-East-2, simply because we cannot get the GPUs that we need, particularly for computational chemistry workloads.

An increasingly attractive alternative, Dagdigian said, is launching clusters in co-location spaces. He highlighted Markley Group as New Englands best-connected co-location facility and gave a quick outline of the solution hes placing there: 1.5 Tb of RAM, 40 CPU cores, four Nvidia Tesla V100 GPUs for about $70,000. As part of a hybrid cloud solution, a co-lo cluster creates a hedge against the cloud GPU scarcity or rising cost. He recommends using such solutions to soak up computational chemistry, simulations, and other persistent workloads.

From his current personal BioTeam workload, Dagdigian is integrating AWS Parallelcluster with Schrodinger computational chemistry tools and SLURMs license-aware job scheduling. It may be a niche use case, he conceded, but an autoscaling HPC grid that understands that you cannot run a job unless a particular license is available is a magic nirvana for us.

Finally, he zipped through a few mini-trends that seem to be at reality inflection points.

Read this article:

Ten Year Trends, Covid's Impact, Where Dagdigian Was Wrong - Bio-IT World