Cloud Hosting

Who, Me? Welcome to a cautionary Who, Me?, a warning to all those lured by the promises of the cloud storage giants and a language lesson for all.

Our story concerns "Dirk", who at the time of our tale was hard at work in a Netherlands IT department. Dirk himself didn't actually speak much Dutch; his first language was English but that was more than enough to get by with in the land of clogs, windmills and dikes.

Having enjoyed a few relatively peaceful months in the job, Dirk told us he was "getting on with tidying up the systems and dealing with the technical debt that had accumulated since the last Big Cleanup a number of years previously."

"One such system," he said, was "cloud-based storage." The vendor was a well-known giant of the industry, but for the purposes of this story, Dirk called it "Poodle"

The task that day was dealing with obsolete accounts, not just from a security standpoint, but also due to licencing costs and, he said ruefully "It seemed like a good idea at the time."

Dirk ploughed through obsolete accounts until he came to one with an odd name: "beheerder". Perhaps an amusing play on "beheader" by some long gone techie? Or something to do with "herding" files? Dirk checked in with his predecessor, who gave the digital equivalent of a shrug. The account had been around since things had been set up back in the day, but nobody had used it in ages.

Indeed, it had been well over a year since anyone had actually logged in using the account. The password was reset and the mailbox scrutinised.

"Perhaps some more notice should have been made of the fact that at one point it was receiving system error messages," sighed Dirk, "but what use is hindsight?"

Anyhow, the messages seemed to have stopped recently and the rest was just junk. To keep things spick and span the account was deleted and, as was standard practice, all files transferred to Dirk.

He thought no more about it until a fortnight later when he decided "to move all those transferred files to the new-fangled Poodle 'Shared Drive'." Heck, everyone else's documents were due to moved there at some point ("with the correct access permission set," he added.)

Dirk clicked "Ignore" on the standard "Files accessed by other users may be affected." This would prove to be a fateful click.

After 20 minutes, one of his colleagues to call to complain that they couldn't access one of their files.

A coincidence, surely. But Dirk began to feel the arse-swooping sensation of dread that something awful might be happening.

Perhaps a little later than he should, Dirk took a closer look at the files being moved. He looked a little closer since some were still in progress. He poked further, down two or three folder levels and saw something that looked familiar.

"Horrifyingly familiar. In fact, it almost looked exactly like the core folder structure that everyone in the office used."

A little more investigation and Dirk realised that he could skip the "almost."

As his stomach sank through the floor, Dirk realised "that over the next few hours file access will disappear for all users."

"It wasn't immediate," he added "because file storage is cloud-based and sometimes it takes its own, undefined, time to do large file moves."

The helpful interface afforded no way to stop the hellish process of borkery. In a panic he called the cloud vendor, observing: "They're rather large and helpful in inversely proportional quantities."

And, of course, they couldn't help. For whatever reason, "there didn't appear to be a way to pause or stop the oncoming slaughter. Everyone was going to lose access to their files."

Having told the increasingly anxious users that there was a "small" problem, Dirk peered at the unfolding carnage he had inadvertently wreaked upon the company's files:

"Some were still in the old location, and some were in the new location."

He tried to move the files again. And again. On the third try he got some response; the system helpfully told him that it was "moving files", but without any kind of countdown of progress bar.

"Empty folders were being created," he told us, "and since the system was still progressing the previous move requests things got very complicated - sometimes files were moving and sometimes not. And sometimes the files were copied before they were moved. On the plus side Poodle Drive would happily create folders before deciding whether or not to copy the files, so that's something isn't it?"

What had happened was that the user "beheerder" had created the root folders back in the day, and other users had created subfolders. A little too late, Dirk discovered that the Move function he had used "only affects folders that you have ownership over. This doesn't include subfolders you didn't create."

"Suddenly," he said "there was a smorgasbord of folders spewn around the system. Some stuff stubbornly in the old location, and some stuff in new locations. Hundreds of gigabytes of data in fact. Smeared everywhere."

The good news was that was a backup, so nothing was actually lost. The only challenge was decide to "Restore" or "Restore with File Permissions." Fearing what would happen to existing files, he restored without those permissions. At least then he'd be able to see all the files and put them back correctly. He could then tidy up the folders and set the correct access levels.

"It took a month," he said.

"Hindsight says that I should have checked out the other option too, but maybe next time(!)"

Our tale ends happily. The users were eventually happy. The storage was tidy. Dirk, while he obviously paid wasn't overtime for his efforts, survived. He also learned to take things a little slower and think a little harder about what those messages were telling him.

Heck, he's even learning Dutch.

Apparently, "beheerder" means "administrator".

Ever been bitten on the behind by cloud storage, or gaily skipped past a message box without fully understanding what it was telling you? Sadly, it is all too common. Share your tale of woe with the sympathetic vultures at Who, Me?

Sponsored: Practical tips for Office 365 tenant-to-tenant migration

Link:
Lost in translation and adrift in cloud storage - The Register

In this era of digital transformation, organizations of all sorts are becoming data-centric information managers. Technologies such as AI, IoT, 5G and edge computing are creating data at unprecedented rates. That data is being used to deliver more insights, services, customized services, products and innovation. And stricter privacy laws and regulations on personally identifiable information with harsh financial penalties for noncompliance are complicating the situation.

For organizations that want to derive value from data that's greater than the cost of storing and using it, effective data management and storage is becoming more important than ever. The abstraction of data management from storage systems to run on its own is one approach to better data management.

Data management has several meanings depending on the vendor. It has been defined as ingesting, storing, organizing and maintaining the data an organization creates. But that definition is outdated. It's adequate as data management for legacy storage systems; however, even that falls short for modern storage systems.

Data management today means considerably more, including:

That's quite a bit for a data management system to do -- and do well. Remember, the most important responsibilities of a data management and storage system are ingesting, storing, organizing and maintaining the data. All those other data management capabilities are resource-intensive and negatively impact the system's primary responsibilities.

These abstracted data management systems have an outsized positive impact on IT organizations.

And most storage systems don't generally work well with other storage systems. That's not news to storage admins. And many systems have problems working with cloud storage, too. Few -- and that's being generous -- actively work with tape systems.

Multivendor heterogeneous storage is a bigger problem. Storage vendors rarely work seamlessly with one another, which is why storage system-centric data management tends to focus on a single vendor. This approach bypasses the multivendor problem while locking in users to that specific vendor's data management and storage products.

Another data management storage issue is the complicated data management licensing structure. There's the data management software licensing, of course, but it doesn't stop there. There are typically other licensing fees, such as a capacity license fee on the data the storage system moves to cloud storage or other storage systems. Then, there's the cloud storage capacity license fee and potentially egress fees for data access. In addition, when users and applications access moved data, it often must be rehydrated back to the originating storage system. Data movement takes time adding substantial latency to each access request. It makes more sense to access the data where it resides.

One storage-centric approach to this problem is to put all an organization's data in a single scale-out, all-encompassing storage system, historically referred to as a "god box." This system would have all the storage performance and cost tiers, data protection, archiving and so on, along with all data management.

Even if that storage system could meet every performance requirement, scale every tier to meet hundreds of petabytes or exabytes of data, and do everything data management needs to do today, there are other intractable problems. The data management software would still be a heavy draw on the storage controllers negatively affecting performance. More importantly, data still must be moved or migrated from where it currently is to this system. And it fails to solve multiorganizational data sharing problems.

These issues have led to a new approach where data management is abstracted from storage systems. The data management software runs on its own server hardware. It sits out of band, in band or a combination of the two.

Abstracted data management captures the data and metadata in one of three ways. It can mount all storage systems with administrative privileges; Dell EMC ClarityNow, Hammerspace, iRODS (open source), Komprise, Spectra Logic StorCycle, Starfish Storage and StrongBox Data Solutions StrongLink do this. Or it can sit in front of the storage looking and acting like a high-speed network switch -- think InfiniteIO here. And with the third approach, it uses clients, or agents, like Aparavi does. Most of these systems have some level of AI or machine learning built into the software that optimizes operations. Each methodology and vendor have their own pros and cons that will be covered in a future article.

These abstracted data management systems have an outsized positive impact on IT organizations. They commoditize the storage system, reducing both the amount and the cost of storage for each tier. They do that by right-sizing the data to the proper tier and eliminating vendor lock-in.

Storage system compatibility is no longer an issue. Storage systems are merely the containers in which the data resides based on their locality, performance and cost characteristics. These abstracted data management systems also simplify operations in several ways:

Each of these data management systems scales differently. Some are designed to scale to hundreds of petabytes and exabytes. Others scale from terabytes to dozens of petabytes. It depends on their architecture, and most are, by definition, storage-agnostic.

Vendors have different licensing requirements. Some license by terabyte of capacity managed. Others vary that capacity licensing by hot and cold data. Still others license according to the number of servers and server cores required to run their software at the performance level an organization requires.

What does external abstracted data management mean? It means IT organizations can choose storage based on cost and performance, not data management functions. It means storage vendors will no longer have an incumbent advantage. It means simplified IT operations and lower costs. When it comes to data management and storage, all that bodes well for the future.

View post:
What the breakup of data management and storage means to you - TechTarget