Cloud Hosting

At the CyberRisk Leadership Exchange in Cincinnati on June 7, the chief security officer of an Ohio bottling company used his lunchtime keynote address to recount how his company's eight-person IT team detected, remediated and recovered from a ransomware attack within the space of seven hours, without losing any business and without paying a dime to the attackers.

"We never missed an order. We never missed a delivery. Customer data was not compromised," Brian Balzer, Executive VP of Digital Technology & Business Transformation at G&J Pepsi-Cola Drink Bottlers, Inc., told SC Media in an interview. "I'd say probably 95% of the organization had no idea that we were under attack."

Balzer credits G&J's rapid, successful recovery from the ransomware attack to diligent preparation and a move to cloud-based operations, and to strong support from both colleagues and company leaders.

"I cannot stress enough to companies the importance of having such a strong culture where people are willing to jump in and help one another," says Balzer, "and a leadership team that is supportive of the cyber team, your IT team, whoever it might be, your digital team, to be able to put plans in motion."

Founded in 1925, G&J Pepsi serves Ohio, Kentucky and West Virginia and is the largest family-owned independent Pepsi bottler in the U.S., with more than a dozen facilities, 2,000 employees and $650 million in annual revenue.

The company first noticed something was wrong with its systems just before the Labor Day weekend of 2021.

"We had suspected that we might have allowed an intruder, a hacker into our environment," Balzer says. "We spent the better part of about four or five days trying to understand where they got in, where they were, if they were in and how we might be potentially exposed."

The G&J Pepsi team suspected that someone had used Cobalt Strike to install "beacons," or backdoors, into the systems, but their initial searches found nothing. Then a few days after Labor Day, a call came in around 4:30 in the morning.

"We got a call from one of the folks in our plant saying, 'Hey, something's weird, I can't access files,'" recounts Balzer. "And we knew instantly that we were under attack."

The G&J Pepsi team quickly took as many systems offline as it could. Balzer credits support from the very top of the company for that.

"I [had] to call my CEO at five o'clock in the morning and say, 'We're literally bringing all the systems down.'" Balzer says. "He's like, 'All right, I trust you. You just keep me posted.'"

Balzer's team found two potential points of entry. The first was a user who had unknowingly downloaded a corrupted file a common vector for ransomware infection but G&J Pepsi's endpoint solution quickly detected and remediated that.

The second point of entry was more serious. Just before the long weekend, Microsoft had released a patch for Exchange Server. But it looked very similar to another Exchange Server patch from two weeks earlier, one that G&J Pepsi had already implemented.

"There was some confusion as to, 'Was this the same patch that they released? Or was this a different patch?'" Balzer recalls. "We just misunderstood. It was probably on us. It was our fault for not getting that clarification quickly."

Instead of implementing the new patch right away, G&J Pepsi decided to wait until the following weekend. That's all the time the ransomware crew identified as Conti by notes left on infected systems needed.

"Within four days, they had exploited that particular gap in that Exchange server, and were able to compromise our environment," Balzer says. But, he added, "we don't deal with terrorists."

As a midsize company, G&J Pepsi fit the profile of a prime target for ransomware crews. The fact that the attack happened over a three-day weekend, which gives attackers more time to operate freely, was likely no coincidence.

"Most midsize companies and small companies can't thwart an attack, particularly a Conti ransomware or other sophisticated attacks that are that are taking place," Balzer says. "When we called for support, we [were told] that 'We'll try to help you, but we are absolutely slammed coming out of this three-day weekend because they went haywire on companies across the U.S.'"

Fortunately, because G&J Pepsi had already moved all its systems to the cloud, shutting down company assets and stopping the attackers was less complicated than it might have been for on-premises infrastructure.

"We have nothing on-premise," Balzer says. "Because we're 100% in the cloud, and because we utilize Microsoft Azure Cloud environment, we were able to prevent them from moving laterally across the platforms in our systems."

The virtual nature of G&J Pepsi's systems meant that the company was able to spend the next few hours using its weekly backups to spin up brand-new Azure instances free of ransomware, even as the team continued to investigate the infected systems.

"Within seven hours, we were able to stand up the entire environment again," Balzer told us. "Many of our solutions are SaaS solutions. The things that were impacted were more like file servers we had a couple of other servers that we had developed as IAS solutions in Azure that were at risk. We were able to basically rebuild and recreate that environment."

G&J Pepsi were lucky. None of the company's backups had been affected by the ransomware, and dark-web scans turned up no evidence of company data having been stolen.

"We were very fortunate that we had eyes on it immediately and were able to basically isolate and wall them off and then rebuild our environment," Balzer says.

However, staff PCs left on overnight in the office were infected, as were some ancillary servers. Rebuilding those took a bit more time.

Following the attack, G&J Pepsi brought on Arctic Wolf as a managed detection and response (MDR) provider and changed several company policies.

"We forced all password resets, we changed our policies on backups, we changed our policies on how many admin accounts that we have we limited those and really revamped the security," Balzer says.

Balzer told us that G&J Pepsi had also locked down USB ports on PCs, beefed up identity and access management and automated its systems patching. As it is a U.S.-only company, G&J Pepsi also blocked all system access from outside the country. The company has not had any serious incidents since.

In a separate interview with Microsoft, G&J Pepsi Enterprise Infrastructure Director Eric McKinney says he has learned two things from the company's brush with ransomware.

"If I could go back in time to the months leading up to our ransomware attack, I'd tell myself to strengthen our endpoint policies," McKinney tells Microsoft. "I don't view our recovery as a victory so much as a call to double down on security."

For McKinney, the second lesson was how much there is to be gained from a full cloud migration.

"G&J Pepsi has gotten a wide range of security benefits, such as platform-based backups, cloud-based identities, and multifactor authentication, leveraging native tools that help recommend and identify risk," McKinney says. "It doesn't matter whether you're a huge corporation like PepsiCo, a midsize business like G&J Pepsi, or a mom-and-pop gas station down the road I would make that move to the cloud and make it quickly."

Fielding questions from the audience following his keynote address at the Cincinnati CyberRisk Leadership Exchange, Balzer was struck by how many of his fellow cybersecurity executives wanted to hear about G&J Pepsi's experience.

"I love that the participation was there, that the curiosity was there," he told us. "People wanted to understand what was happening so that they can be aware of what to do if that ever occurs with them."

But Balzer once more stressed how important company culture is to an organization's ability to maintain resilience and quickly recover from an attack.

"The other thing that really stuck out, that we talked about for a brief bit during that [CyberRisk Leadership Exchange] session," Balzer says, "was the importance of having the right culture within your team to be able to come together to thwart an attack, particularly one of that size or even larger.

"We had a plan in place. Unfortunately, we had to use it. But fortunately for us, that plan worked," adds Balzer. "And that worked because we had the right leadership, the most senior leadership and support, and we had the right culture within our team to help support that and thwart that attack."

The next Cybersecurity Collaboration Forum event in Cincinnati will be a CyberRisk CISO Dinner at the end of September.

For more information on the Cybersecurity Collaboration Forum, including how to attend a CyberRisk Leadership event in your area, please visit https://www.cybersecuritycollaboration.com/.

Many thanks to Zack Dethlefs of the Cybersecurity Collaboration Forum.

See the original post:
How a midsize American business recovered from a ransomware ... - SC Media

When system architects sit down to design their next platforms, they start by looking at a bunch of roadmaps from suppliers of CPUs, accelerators, memory, flash, network interface cards and PCI-Express controllers and switches. And the switches are increasingly important in system designs that have a mix of compute and memory types and for clusters that will be sharing components like accelerators and memory.

The trouble is this: The roadmaps are not really aligned well. Most CPU and GPU makers are trying to do major compute engine upgrades every two years, with architectural and process tweaks in the year in between the major launches so they have something new to sell every year. Makers of chips for networking switches and interface cards in the Ethernet and InfiniBand markets tend to be on a two-year cadence as well, and they used to tie their launches very tightly to the Intel Xeon CPU launch cadence back when that was the dominant CPU in the datacenter, but that rhythm has been broken by the constantly redrawn roadmaps from Intel, the re-emergence of AMD as a CPU supplier, and a bunch of other Arm CPU makers, including at least three hyperscalers and cloud builders.

And then there is the PCI-Express bus, which has been all over the place in the past two decades. And while PCI-Express specifications have been released in a more predictable fashion in recent years, PCI-Express controllers have been faithful to the PCI-Express roadmaps but PCI-Express switches are well behind when it comes to product launches from MicroChip and Broadcom.

Sitting here on a quiet July morning, thinking about stuff, we think all of these roadmaps need to be better aligned. And specifically, we think that the PCI-SIG organization that controls the PCI-Express specification and does so through a broad and deep collaboration with the IT industry, needs to pick up the pace and get on a two-year cadence instead of the average of three it has shown in the past two decades. And while we are thinking about it, we think the industry would be better served with a short-cadence jump to PCI-Express 7.0, which needs to be launched as soon as possible to get I/O bandwidth and lane counts in better alignment with high throughput compute engines and what we expect will be an increasing use of the PCI-Express bus to handle CXL-based tiered and shared main memory.

Dont get us wrong. We are grateful that the PCI-SIG organization, a collaboration between all kinds of companies in the datacenter and now at the edge, has been able to get the PCI-Express bus on a predictable roadmap since the very late PCI-Express 4.0 spec was delivered in 2017. There were some tough signaling and materials challenges that kept the datacenter stuck at PCI-Express 3.0 for seven years, and we think Intel, which dominated CPUs at the time and dragged its feet a little bit on boosting I/O because it got burned with SATA ports in the chipsets used with the Sandy Bridge Xeon E5s that came out later than expected in March 2012. Rumors abounded about the difficulties of integrating PCI-Express 4.0 and PCI-Express 5.0 controllers into processors since then.

Generally, a PCI-Express spec is released and then within about a year or so we see controllers embedded in compute engines and network interface chips. So when PCI-Express 4.0 came out in 2017, we saw the first systems using it coming out in 2018 specifically, IBMs Power9-based Power Systems machines, followed by its use in AMD Rome Epyc 7002s launched in August 2019. Intel didnt get PCI-Express 4.0 controllers into its Xeon SP processors until the Ice Lake generation in April 2021.

And even with the short two-year jump to the PCI-Express 5.0 spec in 2019, it wasnt until IBM launched the Power10 processor in its high-end Power E1080 machines in 2021 that it became available in a product. AMD didnt get PCI-Express 5.0 into a server chip until the Genoa Epyc 9004s launched in November 2022 and Intel didnt get PCI-Express 5.0 into a server chip until the Sapphire Rapids Xeon SPs launched in January 2023.

So it was really a three-year cadence between PCI-Express 4.0 and 5.0 products, as expressed in the controllers on the CPUs, even if the spec did a two-year short step.

We think that the specs and the products need to get on a shorter two-year cadence so the compute engines and the interconnects can all be lined up together. And that includes PCI-Express switch ASICs as well, which have traditionally lagged pretty far behind the PCI-Express specs for the 3.0, 4.0, and 5.0 generations that they were widely available.

The lag between PCI-Express ports and PCI-Express switches at any given generation are a problem. That delay forces system architects to choose between composability (which ideally uses PCI-Express switches at the pod level) or bandwidth (which is provided through a direct server slot). Systems and clusters need to be designed with both composability and bandwidth and we would add high radix to the mix as well.

At the moment, there are only two makers of PCI-Express switches, Broadcom (through its PLX Technologies acquisition a number of years ago) and MicroChip. We profiled the MicroChip Switchtec ASICs at the PCI-Express 5.0 level way back in February 2021, which scale from 28 to 100 lanes and from 16 to 52 ports, but as far as we know, they are not shipping in volume. Broadcom unveiled its PCI-Express 5.0 chip portfolio back in February 2022, including the ExpressFabric PEX 89100 switch, which has from 24 to 144 lanes and from 24 to 72 ports. We are confirming if these are shipping as we go to press and have not heard back yet from Broadcom.

Our point is that PCI-Express switches have to be available at the same time that the compute servers, memory servers, and storage servers are all going to be created using chips that support any given level of PCI-Express. On Day One, in fact. You have to be able to embed switches in the servers and not lose bandwidth or ports or sacrifice radix to get bandwidth. We therefore need lots of suppliers in case one of them slips. This is one of the reasons why we were trying to encourage Rambus to get into the PCI-Express switch ASIC racket recently.

All of this is top of mind just as the PCI-SIG has put out the 0.3 release of the PCI-Express 7.0 spec.

Lets take a look at the projections we did for the PCI-Express roadmap a year ago when the PCI-Express 6.0 spec was wrapped up and PCI-Express 7.0 appeared on the horizon:

The PCI-Express 7.0 spec is not expected to be ratified until 2025, and that means we wont see it appearing in systems until late 2026 or early 2027. We think this wait is far too long. We need PCI-Express 7.0 to provide the kind of bandwidth accelerators need to chew on an enormous amount of data that is required to run a simulation or train an AI model. We need it matched up with a fully complex CXL 4.0 specification for shared and pooled memory.

We understand that it would be hard to accelerate PCI-Express 7.0 controllers and switches to market, and that all manner of products would also have to be accelerated. Compute engine and peripheral makers alike would be hesitant to not try to squeeze as much investment as possible out of their PCI-Express 6.0 product cycles.

Still, as PCI-Express 6.0 is put into products and goes through its rigorous testing which will be needed because of the new PAM-4 signaling and FLIT low-latency encoding that it makes use of we think the industry should start accelerating and match up to the CPU and GPU roadmaps as best as possible and to get onto a two-year cadence alongside of them.

Get the components in balance and then move ahead all at once, together.

Go here to see the original:
PCI-Express Must Match The Cadence Of Compute Engines And Networks - The Next Platform

Joel Loynds

Published: 2023-07-05T13:46:15

Updated: 2023-07-05T13:46:24

The Twitters new CEO, Linda Yaccarino, has broken the tension with new reasoning behind the current reading rate limits, which was also posted on their blog.

After a few days of turmoil behind the scenes at Twitter, Linda Yaccarino, the new CEO of the social media company has come forward and issued a statement, which was coupled alongside a blog post regarding the companys plan with rate-limiting users.

The current owner, Elon Musk, announced on July 1 that the platform would limit how many posts users could see before it locked them from seeing anything else.

Article continues after ad

This led to Twitter essentially DDOS-ing itself and speculation arose around the $1.5 billion bill it might owe in relation to Google Cloud servers.

Yaccarino has broken her silence on the topic, tweeting her support for the action as means of strengthening the platform.

In the tweet, Yaccarino said:

When you have a mission like Twitter you need to make big moves to keep strengthening the platform.

This work is meaningful and on-going.

Ben Collins, a reporter for NBC News questioned the CEO and official blog post:

Subscribe to our newsletter for the latest updates on Esports, Gaming and more.

If this were true, which seems extremely unlikely, why wouldnt you warn users first?

Article continues after ad

Either another lie, or wildly irresponsible, or both.

Meanwhile, the short statement made on Twitters official blog seems to be reiterating what Musk claims is the reasoning behind the limit. The main reason is that people are scraping data from the site without paying for it.

Twitter has recently clamped down on access to its API and began charging for it.

However, recent reports have indicated that since Musks tenure at Twitter, even those paying for the platform have found it broken.

Article continues after ad

Musk has said that the measures are temporary, but weve yet to see the rate limit be alleviated. Meta is currently prepping to launch its competitor, Threads, which launches tomorrow.

Read more here:
Twitters new CEO finally explains rate limits & what it means - Dexerto