Cloud Hosting

Huaweis Alex Agapitos discusses how the latest data science techniques have become necessary in the maintenance of networks.

Data science has become essential in pretty much every industry that uses data, from supply chains and healthcare to insurance and e-commerce.

In the world of telecoms, data science techniques are required to optimise networks through predictive modelling techniques. To learn more about this, SiliconRepublic.com heard from Alex Agapitos, a principal AI architect at the Huawei Ireland Research Centre.

Agapitos has a degree in software engineering and a PhD in computer science. He worked as a post-doctoral researcher in the Complex and Adaptive Systems Laboratory at University College Dublin before joining Huawei in 2016.

He said the introduction of 5G, IoT and edge computing brings new complexities to network operations, which have made manual maintenance infeasible without the latest data science.

Dominant success stories revolve around the use cases of reactive/predictive maintenance and network optimisation, he said.

In the former, outlier detection and predictive modelling techniques mine for patterns in historical data to accurately anticipate and warn about imminent network failures. This allows operators to identify early warning signs of failure and their associated root causes, enabling early interventions before failures affect end users.

Agapitos said another important transformation that data science has brought is autonomous network optimisation.

Deep learning-based predictive modelling allows simulation models of the network environment to be trained using historical data and then combined with data-driven optimisation algorithms that continuously reconfigure the network, he said.

The arrival of data-hungry applications including virtual reality, self-driving cars and gaming will further escalate the need for autonomous data-driven solutions in 5G and beyond.

With data science already driving autonomous network optimisation, Agapitos said he sees an era of intelligent telecommunication networks with minimal human supervision coming down the line.

Advances in multi-agent systems will allow the network to be modelled and implemented as a collection of autonomous agents that perceive their environment and take actions to cooperatively meet a set of global goals, such us keeping the network performance at near-optimal levels at all times, he said.

To deal with ever-changing network conditions, it is crucial for autonomous agents to have the ability to continually acquire, fine-tune and transfer knowledge and skills throughout their life cycle, which is a research area known as continual or lifelong learning.

Advancing lifelong learning for machine learning systems is an ongoing challenge but Agapitos said there is plenty of emerging research in this area.

He also said the advancing complexity and sophistication of intelligent telecommunication networks will inevitably pose a challenge to the human operator in understanding the reasoning behind autonomous decision-making.

Trustworthiness of the autonomous systems internal functionality is of fundamental importance and it will be realised through advances in explainable AI.

Explainable AI is a research area that sits at the intersection of data science, deep learning and symbolic AI. The aim is to develop methods and techniques that produce accurate, explainable models of why and how an AI algorithm or prediction model arrives at a specific decision, so that the result can be understood by a human.

While the need for data grows within society, so too does the question of privacy. Agapitos said he believes the issue of data privacy can be addressed via another area of data science a machine learning technology known as federated learning.

While standard machine learning approaches require centralising the training data in one machine or in the cloud, federated learning enables AI native network elements or user equipment to collaboratively learn a shared prediction model while keeping all the training data on-premise or on-device, he said.

In a nutshell, federated learning proceeds as follows: the network element or user equipment downloads the current model from a shared coordinator, it improves the model by online learning based on data generated locally at the network element or user equipment, and then summarises the model changes as a small model update.

This small update is then sent back to the coordinator using encrypted communication, where it is immediately averaged with peer model updates to improve the shared model. Federated learning allows for smarter models, lower latency, less power consumption, all while ensuring privacy.

Dont miss out on the knowledge you need to succeed. Sign up for theDaily Brief, Silicon Republics digest of need-to-know sci-tech news.

See the article here:

How data science is leading to better network optimisation - Siliconrepublic.com

5 Things to Consider When Operationalizing Your Machine Learning

Operationalizing machine learning models requires a different process than creating those models. To be successful at this transition, you need to consider five critical areas.

As machine learning teams start, most of their work is done in a laboratory mode. This means that they work through the process in a very manual yet scientific manner. They iteratively develop valuable machine learning models by forming a hypothesis, testing the model to confirm this hypothesis, and adjusting to improve model behavior. As these projects mature and evolve, it often becomes important to take them out of experimentation mode and operationalize them.

Operationalizing machine learning requires a shift in mindset and a different set of skills for those performing the work. The inquisitive state of what-ifs and trial and error give way to practices that are predictable and stable. The goal is to reproduce the same valuable results that were generated as part of the creation process but do it in a way that is more hands off and long running. This changes the teams goals from experimentation to experience management.

To effectively operationalize your machine learning model, consider these five key areas: data collection, error management, consumption, security, and model management.

Data Collection

During the experimentation phase, much of the data collection and cleansing is done manually. A training and testing data set is pulled from the source -- that source could be a data lake, a data warehouse, or an operational system -- and is often hand curated. The merging, matching, deduping, and overall data wrangling is generally done one step at a time. This is mainly because the data scientists are not sure what will persist (and what wont) in the data set. This data management process can span from work done in programming languages such as Python and R to work performed using a spreadsheet or a text editor.

With an operational model, the uncertainty of what data is valuable is removed and all the data wrangling done during the build phase now needs to be automated and productionalized. This means that the scripts used during the development phase need to be standardized into something that can be supported in a production environment. This can mean rewriting scripts into a supported language, automating the steps performed in a spreadsheet using scripting or an ETL tool, or ensuring that all the data sources used are being updated regularly and are accessible as part of the data collection process.

Error Management

When data scientists are working through the process one step at a time, they manage the errors that arise. From dirty data to data access issues, if data scientists run into a problem, they interact with the people and systems that can resolve it. With these unanticipated challenges, the most effective path forward is to address them one at a time as they arise.

This is not the case once the models have been promoted to a production environment. As these models become integrated with an overall data pipeline, downstream process come to rely on their output and errors have a higher risk of business disruption. As many of these potential errors as possible need to be anticipated during the pre-operation design and development stage and automated mechanisms need to be designed and developed to address them.

Read this article:

5 Things to Consider When Operationalizing Your Machine Learning | Transforming Data with Intelligence - TDWI

Data science has the power to accelerate social and environmental

progress. Yet according to Salesforce.orgs Nonprofit Trends Report, only 22% of social impact organizations have achieved high data maturity today. As a result, the data for good sector has the tendency to rely too heavily on creating flashy, new tools to fix problems. But these tools often fail to move the needle on real impact, and many underserved communities and the non-profit organizations that serve them continue to need better access to skills and capacity to leverage these innovations.

social and environmental progress. Yet according to Salesforce.orgs Nonprofit Trends Report, only 22% of social impact organizations have achieved high data maturity today. As a result, the data for good sector has the tendency to rely too heavily on creating flashy, new tools to fix problems. But these tools often fail to move the needle on real impact, and many underserved communities and the non-profit organizations that serve them continue to need better access to skills and capacity to leverage these innovations.

In order to actually deliver impact through data science at scale, what needs to change across our sector?At a recent data.org event, we convened social impact organizations, funders, and data science leaders to explore ways to address this challenge. We sought participants insights and gained a clearer sense of what it will take for data to be accessed and applied for good.What follows are three calls to action that emerged from our conversation. We believe that realizing these calls would catalyze a shift toward scalable, sustainable, and genuinely community-driven projects that help the social good sector use data science to realize impact.

Its easy to fall for the flash and glimmer of a new AI solution but we cant stop there. We have to deepen our understanding of the problems that we are trying to solve, and our commitment to working with the people and communities that experience real challenges everyday. This might seem like a small shift, but its seismic. It pushes us beyond thinking only about the mechanics of a technical solution and instead challenges us to ask how new technology can change the balance of power in favor of people and communities that have been systematically excluded or harmed. To be clear, passion for new technical solutions isnt bad. Many problems we face in the social impact sector do require innovation and creativity. But simply having a new approach doesnt guarantee actual impact. Our metric for success cannot simply be that we delivered a solution. That solution must meaningfully contribute to reducing suffering or improving equity.

Doing this isnt easy. It requires technical experts to diversify their networks and engage with humility. True understanding of social issues cannot be done without community experience and partnership. Creating technology far from the community it purports to benefit rarely works. Instead, we must partner with communities to develop solutions that are responsive and designed to scale in the real world.Funders play a critical role in shifting the focus from novel solutions to actual impact. Much of the innovation funding ecosystem currently focuses on building new things instead of investing in long-term capacity building and problem solving. As solution builders, it can be easy to lose focus on the impact you seek in favor of amplifying what will be most attractive to funders. Change makers and funders bear a joint responsibility to honor the complexity and context of the problem at hand and continually seek to deliver impact, not getting distracted by a desire to over index on what might be considered the shiny, data-driven technology of the moment. Disciplined focus on what specific problem data science is helping you understand or address at any one moment in time is essential when unlocking the power of this technology. Without a disciplined approach, the use of data science can be distracting and potentially dilute or derail your impact.So, we must follow the problem. And one of the things we might learn as we follow it is that the problem is not solvable because of a single data science method. For people coming from data science backgrounds and engineering backgrounds, that means that you might actually have to admit that you maybe arent the biggest part of the solution. And that reflection, and the maturity around that reflection, is absolutely critical for figuring out what you can do, for figuring out an angle in, for figuring out an approach or an impact model that actually does speak to the real problem. You have to identify what problem it is that you are capable of solving and find true product-impact fit. While following the problem seems intuitive, it is inherently very difficult. But its urgently necessary if we want to advance and truly use data to drive impact rather than just giving rise to pilots that explore emerging technologies. As social impact officers, implementers, and funders, we must honor the complexity of the problems that we seek to solve, and be committed enough to fall in love with the actual problems themselves.

Advancing our sector also means seeing and supporting projects through to the very end, to where people are applying it to their everyday lives or organizations. It is much easier to build a new product and get it to a Minimal Viable Product stage. But then, to deliver on the impact, you have to actually use the product over time. You have to build the muscle for iteration. Embracing iteration helps to solve one key challenge social impact organizations face: a lack of clarity around the metric for which they are optimizing. In profit-driven business, its much more straightforward: Does a new recommender algorithm, for example, increase engagement, conversions, and then revenue?But for social impact organizations, measurement and agreement on what the key metrics actually are can make this messier. Building a muscle for iteration means you commit to actually looking at the outcomes of deploying a new method, and that youre able to regularly and reliably measure those outcomes at a reasonable cost. And like building muscle in the gym, this process requires trial and error and an ongoing commitment.Funders have traditionally taken a very linear, more short-term approach to supporting solutions providing resources to get to the end of an initial pilot, for instance but the messy nature of achieving impact goals demands that we should be embracing a more iterative mindset and approach. Common case studies for success like BlueConduits data driven approach to helping Flint with its water crisis or GiveDirectlys efforts to use data science to target cash transfers for COVID-19 relief all reflect an iterative narrative, reinforcing the ideal process of idea, implementation, and success, with funding and governmental support at every step of the journey. However, those seamless journeys are the exception, not the rule. The reality of driving impact outcomes is more like life: unpredictable and requiring constant course correction. Imagine an exciting new algorithm that promises to solve hunger in a community. We might expect there to be funding to build the algorithm, have the paper written about it, get press published; but, when it comes to working through the application of it with 20 non-profits with different use cases, we may realize that the algorithm will need continuous refining, and that the exercise of testing and refining will take us in new and unexpected directions around how to effectively serve diverse neighborhoods or, at worst, that no one needs the technology in its initial form, and well have to go back to the drawing board and build something fundamentally different from the initial solution.Thats where our current systems for funding and support can fall apart. So, we need solution builders and funders to anticipate and embrace the 2.0s of the project, the 3.0s, and beyond. Only through the creation of Minimum Viable Products and its testing phase can we understand that component of the problem statement that we can effectively influence, improve, predict, or make more efficient.

Sustaining and scaling data science for impact requires a deep commitment to capacity building and technical education. This capacity building must happen across the ecosystem, from implementing organizations, through to funders. At this stage investing in the capacity of humans is probably the most powerful thing that we can do to move along the transformation curve. Because humans and systems are what actually move the needle on solving problems, investments in human systems ensure that innovation happens at scale, rather than just one thing at a time. Katharine Lucey, who leads Solar Sister, is a perfect example of what you unlock when you invest in the humans and internal capacity behind a solution. With data.orgs support through the Inclusive Growth and Recovery Challenge, she invested in making sure she had data experts on her team and the budget to support them in the long term. As a result, her work in supporting local women entrepreneurs in Africa who work with clean energy has become a model for how data science can help steer social impact. That evolution is the direct result of investments in capacity. As another example of building capacity of partners: The Center for Global Action devises a system for locating and measuring poverty. But the step that actually helped people in poverty was getting money to them, and having policy makers who understood this system and could adapt it and move it through. So the CEGA system of data measurements for poverty was important, but only in as much as it enabled a sophisticated, human-driven administrative process that was actually distributing money.At the end of the day, it will be our subject matter experts who understand the complexity and the context of the challenges faced by the communities seeking to solve problems in their neighborhoods. We have a responsibility to make sure that this type of thinking, learning, and tooling is available. How do we train more? How do we implement more for more people?As problem solvers, and funders of problem solvers, there needs to be more consideration of the patience of capital especially when were talking about product-impact fit and learning around how to fund product roadmaps. We need to be asking not just, What can the technology do? but, How do we train more people? How long can they sustain this work? What else do the people doing this work need? How do we build interdisciplinary teams that have the data skills, technical skills, community insight and subject matter expertise of the problem?

Funders or impact partners shouldnt be afraid if any of this sounds overly ambitious or daunting: its just a different mindset, and different set of knowledge to acquire. We can all do this together but to do it, we must change how we build, fund, train, support, and lead the sector moving forward. We must move from being solutions-focused to being problem-focused, from launch-focused to iteration-focused, and from tech-focused to capacity-focused. These challenges require all of us innovator, funder, and implementer alike to contribute. Theyre complex challenges, but its exactly what data.org was set up to do. For practical information and inspirational ideas to help social impact organizations use data science to solve the worlds biggest problems, check out data.orgs public resource library.

View post:

From "data for good" to "data for impact" - NationSwell

Utilities canand MUSTplay a leading role in the clean-energy transition, and in their own sustainable transformation.

Depending on how you look at it, the glass is either half full or half empty. The half full view says that if utilities can rise to the challenge and meet the 2030, 2040 and 2050 sustainability goals they have laid out, worldwide emissions could fall precipitously. The half empty response counters that the utility industry is constrained by regulation that stifles innovation. Adding to the difficulty, the utility industry is constrained by downward cost pressure, meaning large rate hikes are not an option.

As someone who has dedicated 35+-years to a career in the utility space and seen the best and worst tendenciesand everything in betweenin our industry, I may surprise you by saying Im not in the half-full camp. I think theres even more in the glass. The challenge ahead is likely the biggest weve ever faced as an industry, but it is also a once-in-a-lifetime opportunity to grow, radically improve, and make a lasting positive impact on society, all fueled by the following macro-drivers:

Sustainability: Decarbonization efforts in service of reaching net zero carbon emissions are the clarion call of the industry. Mass adoption of renewables at scale wind, solar, battery storage, and an emerging set of new energy sources is in motion. At the same time, collaborating with business and residential customers using distributed energy resources, such as solar and demand response, will only grow in importance. Feeding into all of this is increasing demand for EVs, part of an electrification wave that could drive utility growth for decades.

Safety and Reliability: While utilities are fighting the consequences of climate change more severe fires, droughts, floods, and temperature extremesthey are doing so from a position of weakness, and with aging infrastructure. A smarter, more resilient grid must be built to literally weather the storm, and it will cost billions.

Equity: It has always been an issue, but because of COVID-19 and its ongoing variants, low and moderate income (LMI) residential customers and small and medium sized businesses (SMBs) have gotten clobbered, and they need, and are demanding, more from their utilities to handle energy expenses and gain access to sustainability programs.

But we cant just spend our way out of this. The limited ability of utilities to raise rates fundamentally changes the math of transformation. With hundreds of billions in needed capital investment on one side and affordable rates on the other, O&M costs have moved to the middle, becoming the fulcrum. Simply put, utilities numbers for sustainability, safety, reliability, and equity dont and cant add up without a massive decrease in O&M expense and optimized capital expenditures.

But how is that going to happen? That is literally the multi-billion-dollar question, one that necessitates a fundamentally new-and-improved approach. That approach is rooted in data. The strategic use of data is now a must-have capability for every utility in search of increased customer engagement, infrastructure optimization, and sweeping cost reduction.

Just look at the Sustainability Transformation Plan laid out by Evergy. The plan accelerates Evergys transition to cleaner energy (80% reduction in CO2 emissions by 2050), increases capital investment in critical utility infrastructure to nearly US $5 billion, anticipates 5-6% rate base growth and 6-8% EPS through 2024, and promises increased benefits for customers and communities. And it reduces O&M costs by 25%.

Thats thinking BIG.

But theres a reason this big thinking is more than wishful thinking. The prevalence of AMI and other utility and customer data, the maturation of predictive data science, the evolution of AI and machine learning solutions, and the growing understanding of how to put them all together to move the needle quickly and efficiently means there has never been a better time for utilities to play offense.

Instead of a cadence-based approach to managing the grid, the industry is moving to a data-driven risk-spend efficiency approach to optimize capital investments and O&M spend, managing reliability proactively at a more granular level. In addition to cost savings, this data-driven approach also enables a more collaborative regulatory process I call results-based compliance, which ties spending directly back to reliability measures. It relies on data, not a calendar, to optimize spending on vegetation management, for example.

With data in hand, the time is NOW for utilities to dive in and accelerate their own transformation to becoming a more sustainable utility.

Because utilities matter. They always havejust ask anyone without power or safe drinking waterbut they matter now more than ever. In addition to keeping the lights on, the coolers cooling, and the water flowing, utilities are now poised to play a starring role in the very survival of the planet.

The good news is its doable, and doable now.

Ted Schultz is the chief executive officer of ESource. He has more than 35 years of industry experience and is known for his focus on providing value to customers. Prior to joining ESource, he served as CEO of TROVE Predictive Data Science, senior vice president of utility solutions at Ecova, and vice president of marketing and energy efficiency at Duke Energy. Ted has held several advisory board positions at the national level, including with the Edison Foundation, National Action Plan for Energy Efficiency, EPRI Energy Services, and JD Power Smart Grid Services. He currently serves on the Marketing Executives Conference.

Continue reading here:

Data, Decarbonization, and The New Math of Sustainability - Transmission & Distribution World