Seattle More than 2 billion people log into Facebook every month. Every day, the social-media crowd uploads billions of photos, calls up hundreds of millions of hours of video, and fires off a prodigious flurry of likes and comments. Somebody has to store that slice of humanitys digital record.
Much of that task falls to Surendra Verma, a Seattle engineer who for more than 20 years has been building software that files away and retrieves large volumes of data.
Verma leads the storage team at Facebook, the group charged with making sure that the social network can accommodate its daily deluge without losing someones wedding photos.
Most of that unit is based in Seattle, part of a workforce that today numbers 1,600 people, up from just 400 three years ago. That makes Facebook one of the fastest- growing technology companies outside of Amazon, anyway in the city.
While Facebook employees work on a wide range of products in Seattle, the office has developed a specialty in the geeky realm of systems software.
About a quarter of the Facebook engineers in Seattle work on the companys infrastructure projects, the tools to transmit, store and analyze the growing heap of data people feed into the social network.
Thats a common trade in the region, where Amazon Web Services, Microsoft and Google are all building their own clouds giant, globe-straddling networks of data centers and the software that manages them.
Facebook could have built its products on computing power rented from those cloud giants, but it decided to build its own tools, from custom hardware designs all the way to mobile applications. Supporting Facebooks network are nine massive data centers a 10th, in Ohio, was announced earlier this month.
Facebooks cloud is different from the others in that its designed to support just one customer: Facebooks own apps.
They happen to be some of the most widely used pieces of software in the world and their use keeps expanding.
Verma is an Indian-born engineer who got his start at IBM before moving to Microsoft, where he worked most recently on Windows file systems. He joined Facebook in 2015.
By then, the Seattle office had come to pilot Facebooks storage-software efforts. We could find some very good engineers here, he said. And so a bunch of these projects started, and just got momentum from there.
His teams job, he said, is to be invisible to Facebooks product groups, letting the companys other engineers build whatever they can think up on top of a reliable foundation.
But a wave of new services, and the rapid growth in the number and quality of videos and photos that people share, is putting a huge burden on the networks infrastructure.
Facebook in January 2016 started a monthslong rollout of live video streaming, a milestone in the social networks effort to compete with popular streaming services.
Then Instagram Stories, aimed at competing with the ephemeral photo-sharing application Snapchat, launched last August, and quickly made its way to 250 million monthly users. (Facebook bought Instagram in 2012.)
Up next: live video broadcasts on Instagram, called Live Stories, a feature the product group hoped to launch before Thanksgiving.
Vermas team wasnt sure its systems could meet that deadline, and negotiated a few days of delay on the planned start date. After scrambling, the team scraped together the internal storage space needed to accommodate the new feature, which went live Nov. 21.
We looked very hard at our capacity, everywhere, Verma said. We scrounged everything we could, looked at nooks and crannies.
One of the people doing the looking was J.R. Tipton, a manager on Vermas team.
Tipton left the Chicago suburbs and came to Seattle in 2001 for the same reason as thousands of others in the 1990s and 2000s: a job offer from Microsoft.
Fifteen years later, he became part of another phenomenon reshaping the region, opting to leave maturing Microsoft for a job with a younger tech giant in the area.
I wanted an adventure, Tipton said of his move to Facebook last year.
Tipton and Verma are among the 640 people at Facebook in Seattle who, on LinkedIn, list past experience at Microsoft.
Seattle, home to Boeings legions of engineers long before Microsoft came along, has never really been a one-company town. But the range of options for technologists has ballooned in the last decade, with Amazon uncorking another Microsoft-like growth spurt, and Silicon Valley giants seeding local outposts to scoop up some of the talented software developers here.
When Facebook set up shop in Seattle in 2010, it was the companys first U.S. engineering office outside its Menlo Park, Calif., headquarters. Last year, Facebook Seattle, which then numbered about 1,000, moved into nine floors of custom-designed office space. That was followed in short order by two more leases that would give the company enough space for about 5,000 employees in Seattle.
Today, Tipton works on the Facebook system that is the last line of defense keeping peoples photos and videos from getting wiped out by a software error or power outage at a data center.
That would be cold storage, the backup system storing redundant copies of all of the photos, videos and other data Facebook stores.
The engineers who designed the system in 2013 wanted to build something more efficient than the typical rack of servers, which suck up energy both to keep their hard drives powered on and to run the fans and air-circulation systems that keep the stack of electronics from overheating.
The company landed on a design that leaves most hard drives in Facebooks rows of servers powered down until needed, hence the cold in cold storage. The software built to run that system was designed to handle plenty of growth in the amount of data people were throwing at Facebook.
Each cold-storage room in Facebooks data centers was built to house up to one exabyte of data. Thats 1,000 petabytes, or 1 billion gigabytes, or the combined hard drive space of 500,000 top-of-the-line MacBook Pro laptops. At the time, it seemed laughably huge, Tipton said of the software layer built to manage all of that.
Last year, it became clear it wouldnt be enough, the result of growth in both the number of Facebook users and the launch of data-intensive features like Instagrams Live Stories.
Tipton and his Seattle-based group spent the first half of 2016 taking a long look at the cold-storage software, which was starting to show its age in a world of 360-degree video. They could invest more time and energy in keeping the old architecture working, or rebuild.
They opted to rebuild.
It has to be much, much bigger, he said.
Tipton has metaphors at the ready to describe that continuing challenge, the fruit of years explaining his day job to nontechies.
You stand on top of a hill and see the buffalo herd coming, he said.
And if that doesnt sound harrowing enough, he tries again: Were basically laying the railroad track down as the train is coming.
Here is the original post:
Digital Deluge on the Cloud – Valley News
- Brinkster Cloud Servers - VMware, SolidFire SSD-Based ... - June 2nd, 2018
- Virtual Network Virtual Private Cloud | Microsoft Azure - March 29th, 2018
- Keeping Your Files Safe in Google's Cloud - New York Times - September 7th, 2017
- 5 Reasons SD-WAN, 4G LTE Are Cloud Essentials - No Jitter - September 7th, 2017
- Canon USA Advances PRISMAsync Color Print Server in Version 5.2, Offering Cloud-Based PRISMAlytics Dashboard ... - PR Newswire (press release) - September 7th, 2017
- HPE Reports Q3 Gains Along With Cloud Deal - EnterpriseTech - September 6th, 2017
- Huawei Releases the New-Generation Intelligent Cloud Hardware Platform Atlas - Markets Insider - September 6th, 2017
- Unlocking the promise of a connected world through edge cloud ... - ITProPortal - September 5th, 2017
- Want to do IoT right? You'll need more storage, networking, servers, and cloud - TechRepublic - September 5th, 2017
- So you're already in the cloud but need to come back down to Earth - The Register - September 5th, 2017
- Nasa: Our demands for repeat presidential election - Daily Nation - September 5th, 2017
- Chinese smartphone maker Xiaomi open to moving servers to India - Economic Times - September 5th, 2017
- VMware officially lands on AWS cloud with new management and security features - SiliconANGLE News (blog) - September 2nd, 2017
- VMware-on-AWS is live, and Virtzilla is now a proper SaaS player - The Register - September 2nd, 2017
- Socionext Partners with Advantech to Offer High-Density, Low-Cost ... - Design and Reuse (press release) - September 2nd, 2017
- Municipal adoption of the cloud - American City & County (blog) - August 31st, 2017
- Veeam follows Virtzilla's cloud up the Amazon - The Register - August 31st, 2017
- Where does a business's data live? - Information Age - August 31st, 2017
- IBM cooks up a hardware architecture for tastier cloud-based services - TechTarget - August 31st, 2017
- Tachyum bets on flash storage to re-architect the cloud data center - ZDNet - August 29th, 2017
- Juniper adding microsegmentation to Contrail cloud - TechTarget - August 29th, 2017
- The future of serverless cloud looks a lot like physical servers - TechRepublic - August 29th, 2017
- Demand for server specialists increases, but talent pool is small - Network World - August 29th, 2017
- The pros and cons of cloud vs in house servers - Edmonton - August 28th, 2017
- You Can Now Spin Up VMware Servers in Amazon Data Centers - Data Center Knowledge - August 28th, 2017
- Windows Server 2016 changes prompt a new look at management - TechTarget - August 28th, 2017
- Cloud security market to reach $12B by 2024, driven by rise of cyber attacks - TechRepublic - August 28th, 2017
- Jeff Pulver, Internet Pioneer of VoIP and Entrepreneur Joins ... - Markets Insider - August 28th, 2017
- Google Aims to Boost Cloud Security with Titan Chipset - BizTech Magazine - August 28th, 2017
- Oppo and Vivo plan to move cloud storage to India, following India's new directives on data security - Firstpost - August 28th, 2017
- How Can You Improve Document Management By Integrating Cloud-Based File Sharing And What You Need To Know ... - Business 2 Community - August 27th, 2017
- Hitachi rack servers get VMware Cloud treatment The Register - The Register - August 26th, 2017
- CenturyLink enhances VMware-based DCC platform, touts software-defined data center approach - FierceTelecom - August 26th, 2017
- Biz sends apps to public cloud, waves 'bye to on-premises server ... - The Register - August 23rd, 2017
- Druva Raises Another $80 Million - Channel Partners - August 23rd, 2017
- CrashPlan alternatives: How to move to another home backup solution - Macworld - August 23rd, 2017
- VMware shares to surge more than 20% because the Amazon cloud threat is overblown: Analyst - Yahoo Finance - August 23rd, 2017
- AMD Lines Up New China Datacenter Partners - EnterpriseTech - August 23rd, 2017
- How do you bring artificial intelligence from the cloud to the edge? - TNW - August 21st, 2017
- The rice of cloud, avocado of virtualization and salmon of doubt: Let's eat storage sushi - The Register - August 21st, 2017
- 70% of firms face skill shortages for server-based roles - Cloud Pro - August 21st, 2017
- Qualcomm moved its Snapdragon designers to its ARM server chip. We peek at the results - The Register - August 21st, 2017
- Info on 1.8 million Chicago voters exposed on Amazon server - USA TODAY - August 21st, 2017
- Microsoft and Google Give Startups Options to Amazon's Cloud - Fortune - August 18th, 2017
- Cloud is the ignored dimension of security: Cisco - ZDNet - August 18th, 2017
- How AIG moved commercial claims to the cloud - Information Management - August 18th, 2017
- Oracle expands database offering to its cloud services - Network World - August 16th, 2017
- Voices Cloud security from all angles - Accounting Today - August 16th, 2017
- HostHatch launches new Cloud Servers - 5x faster than the giants, including AWS & DigitalOcean - PR Web (press release) - August 15th, 2017
- Oracle Exadata Cloud lands on bare-metal servers - Computer Business Review - August 15th, 2017
- School phones go on 'the cloud' - The Ridgefield Press - August 15th, 2017
- Datrium Announces Split Provisioning For Simple Private Cloud Consolidation At Rackscale - Markets Insider - August 15th, 2017
- New McAfee virtual network security platform offered as part of free test drive on Amazon Web Services - CTR - August 14th, 2017
- How to move into a cloud career from traditional IT - InfoWorld - August 14th, 2017
- Oracle Makes the Most Powerful Database Platform Available on the Industry's Most Advanced Cloud Infrastructure - PR Newswire (press release) - August 14th, 2017
- Frank Dinucci's Cloud Accounting Workshop Draws Many ... - Markets Insider - August 13th, 2017
- Frank Dinucci's Cloud Accounting Workshop Draws Many Entrepreneurs - PR Newswire (press release) - August 12th, 2017
- Cryptocurrencies have pulled one of Nvidia's most sluggish businesses out of the gutter - Quartz - August 11th, 2017
- GoDaddy tops Q2 targets, revenue up 22 percent - ZDNet - August 9th, 2017
- Hardware Can Still Make or Break the Cloud - IT Business Edge (blog) - August 9th, 2017
- Serverless Architectures from an MSP's Point of View - MSPmentor - August 4th, 2017
- Unisecure Data Centers Offers 15% Discount On Cloud Server Hosting Services - HostReview.com (press release) - August 2nd, 2017
- How The Cloud Will Disrupt The Ad Tech Stack - AdExchanger - August 2nd, 2017
- Packet launches edge compute service in 15 global locations - RCR Wireless News - August 2nd, 2017
- IBM adds Optane to its cloud, only as storage and without GPUs - The Register - August 2nd, 2017
- Joining Apple, Amazon's China Cloud Service Bows to Censors - New York Times - August 1st, 2017
- Cisco Launches New UCS Servers, Hybrid Cloud Management ... - SDxCentral - July 12th, 2017
- Verizon data of 6 million users leaked online - CNNMoney - July 12th, 2017
- Server vendors board the Xeon SP party bus - The Register - July 12th, 2017
- New Azure servers to pack Intel FPGAs as Microsoft ARM-lessly embraces Xeon - The Register - July 12th, 2017
- Hybrid cloud and blockchain solutions will be the future for data backup - Information Age - July 10th, 2017
- New 'Microsoft 365' package bundles Windows and Office for businesses - GeekWire - July 10th, 2017
- Tech Data Tightens Cloud Integration With Microsoft To Unlock Simpler Experience For SMBs - CRN - July 10th, 2017
- 6502 Retrocomputing Goes to the Cloud - Hackaday - July 8th, 2017
- Retail apocalypse may jump start suburban renewal - Lexington Herald Leader - July 8th, 2017
- Evaluating AMD's Server Market Potential - Seeking Alpha - July 7th, 2017
- Data Center Security: How Cloud Services Keep Your Files Safe - Cloudwards - July 7th, 2017
- Cubic Subsidiary Gets NSA Clearance for Cloud Servers With Aruba Virtual Mobility Controller - ExecutiveBiz (blog) - July 7th, 2017
- As cyberattack hit, Ukrainians turned to Facebook and Google - Houston Chronicle - July 6th, 2017
- Baidu Deploys Xilinx FPGAs in New Public Cloud Acceleration ... - Design and Reuse (press release) - July 6th, 2017