Cloud Hosting

The rise of self-service BI tools enabled people outside of IT to analyze data and create data visualizations and dashboards on their own. That was terrific when the data was ready for analysis, but it turned out that most of the effort in creating BI applications involved data preparation. It still does -- and numerous challenges complicate the data preparation process.

Increasingly, those challenges are faced by business analysts, data scientists, data engineers and other non-IT users. That's because software vendors have also developed self-service data preparation tools. Those tools enable BI users and data science teams to perform the required data preparation tasks for analytics and data visualization projects. But they don't eliminate data prep's inherent complexities.

In the modern enterprise, an explosion of data is available to analyze and act upon to improve business operations. But the data used in analytics applications is often gathered from various sources, both internal and external. Most likely, it is formatted in different ways and contains errors, typos and other data quality issues. Some of it may be irrelevant to the work at hand.

As a result, the data must be curated to achieve the levels of cleanliness, consistency, completeness, currency and context needed for the planned analytics uses. That makes proper data preparation crucial. Without it, BI and analytics initiatives are unlikely to produce the desired outcomes.

Data preparation has to be done within reasonable limits. As Winston Churchill said, "Perfection is the enemy of progress." The goal is to make the data fit for its intended purpose without getting stuck on analysis paralysis or endlessly striving to create perfect data. But it can't be neglected or left to chance.

To succeed, it's important to understand the challenges that data preparation presents and how to overcome them. Many data preparation challenges could be bundled together under the data quality label, but it's useful to differentiate them into more specific issues to help identify, fix and manage the problems. With that in mind, here are seven challenges to be prepared for.

Data analysts and business users should never be surprised by the state of the data when doing analytics -- or worse, have their decisions be affected by faulty data that they were unaware of. Data profiling, one of the core steps in the data preparation process, should prevent that from happening. But there are different reasons why it may not do so, including the following scenarios:

How to overcome this challenge. Solid data profiling needs to be the starting point in the data preparation process. Data preparation tools can help with that: They include comprehensive data profiling functionality to examine the completeness, cleanliness and consistency of data sets in source systems and then in target ones as part of data curation. Done well, data profiling provides the information needed to identify and address many of the data issues listed in the subsequent challenges.

A common data quality issue is fields or attributes with missing values, such as nulls or blanks, zeros that represent a missing value rather than the number 0, or an entire field missing in a delimited file. The data preparation questions raised by these missing values are whether they indicate that there is an error in the data and, if they do, how should that error be handled. Can a valid value be substituted in? If not, should the record (or row) with the error be deleted, or kept but flagged to show there's an error?

If they aren't addressed, missing values and other forms of incomplete data may adversely affect business decisions driven by analytics applications that use the data. They can also cause data load processes that aren't designed to handle such occurrences to fail. That often results in a scramble to figure out what went wrong and undermines confidence in the data preparation process itself.

How to overcome this challenge. First, you need to do data profiling to identify data that's missing or incomplete. Then determine what should be done based on the planned use case for the data and implement the agreed-upon error handling processes, a task that can also be done with a data preparation tool.

Invalid values are another common data quality issue. They include misspellings, other typos, duplicate entries and outliers, such as wrong dates or numbers that aren't reasonable given the data's context. These errors can be created even in modern enterprise applications with data validation features and then end up in curated data sets.

If the number of invalid values in a data set is small, they may not have a significant impact on analytics applications. But more frequent errors may result in faulty analysis of the data.

How to overcome this challenge. The tasks to find and fix invalid data are similar to the ones for handling missing values: Profile the data, determine what to do when errors are encountered and then implement functions to address them. In addition, data profiling should be done on an ongoing basis to identify new errors. This is a data preparation challenge where perfection is not likely to be attained -- some errors will inevitably slip through, but the intent should be to do whatever it takes to keep them from adversely affecting analytics-driven decisions.

One more data quality issue that complicates data preparation is inconsistency in the names and addresses of people, businesses and places. This type of inconsistency involves legitimate variations of that data, not misspellings or missing values. But if not caught when preparing the data, such inconsistencies can prevent BI and analytics users from getting a complete view of customers, suppliers and other entities.

Examples of name and address inconsistencies include the following:

How to overcome this challenge. The source data schemas must be examined to determine what name and address fields are included, and then the data profiled to identify the scope of the inconsistencies. Once you've done that, the following are the three optimal ways to standardize the data:

Inconsistent data also is often encountered when multiple data sources are needed for analytics. In this instance, the data may be correct within each source system, but the inconsistency becomes a problem when data from different sources is combined. It's a pervasive challenge for the people who do data preparation, especially in large enterprises.

How to overcome this challenge. When the data inconsistency is the result of an attribute such as an ID field having different data types or values in different systems, data conversions or cross-reference mapping can be used for a relatively easy fix. However, when it occurs because business rules or data definitions are different across the source systems, analysis must be done to determine data transformations that can be implemented while preparing the data.

One of the key steps in creating the business context needed for analytics is enriching data. Examples of data enrichment measures include the following:

But enriching data isn't an easy task. Deciding what needs to be done in a data set is often complicated, and the required data enrichment work can be a time-consuming procedure.

How to overcome this challenge. Data enrichment should start with a strong understanding of the business needs and goals for analytics applications. That will make it easier to identify the business metrics, KPIs, augmented data and other enrichments required to meet those needs, and then to define things like filters, business rules and calculations to generate the enriched data.

Although data scientists and other analysts perform many ad hoc tasks, the more impactful data preparation work they do inevitably becomes a recurring process that then expands in scope as the resulting analytics becomes more and more valuable. But organizations often encounter problems with that, especially if they're using custom-coded data preparation methods.

For example, what happens and why in a data preparation process is typically known only by the person who created it if there's no documentation of the process or of data lineage and where data is used. The dependency on such individuals requires them to spend increasingly more time on these processes and makes it hard to sustain the data preparation work when they leave the organization.

In addition, when changes or enhancements to a data preparation process are needed, bolting on new code makes the process more precarious and difficult to maintain.

How to overcome this challenge. Data preparation tools can help you avoid these traps and achieve long-term, sustained success in preparing data. They provide productivity and maintenance benefits such as pre-built connectors to data sources, collaboration capabilities, data lineage and where-used tracking and automated documentation, often with graphical workflows.

To succeed at data preparation, it's imperative that you first understand what data is needed for an analytics application and the associated business context. Once the relevant data has been gathered from source systems, the key steps in preparing it include the following:

As you go through those steps, do what's appropriate and possible in a reasonable way, especially in cleansing the data. Keep in mind that perfection often isn't attainable or may not be worth the cost to achieve -- and that it really can be the enemy of progress on data preparation.

Read this article:

Top data preparation challenges and how to overcome them - TechTarget

The University of Helsinki is the oldest and largest institution of academic education in Finland, an international scientific community of 40,000 students and researchers. In international university rankings, the University of Helsinki typically ranks among the top 100. The University of Helsinki seeks solutions for global challenges and creates new ways of thinking for the best of humanity. Applicants of any gender, linguistic and cultural background or from members of minority groups are welcomed.

The Faculty of Social Sciences is Finlands leading research and education institute in the social sciences, and the countrys most diverse social sciences faculty in terms of the range of its disciplines and fields of research. The Faculty has a strong international research profile and an international masters programme, and several of its disciplinary units have been rated among the top 50 in the world. The Faculty has approximately 500 research and teaching staff, and each year it awards some 350 bachelors degrees, 400 masters degrees and more than 40 doctoral degrees. For more information, please see the Faculty website at http://www.helsinki.fi/en/faculty-social-sciences.

The Faculty of Social Sciences and the Centre for Social Data Science invite applications for the position of

UNIVERSITY LECTURER IN DEMOGRAPHY AND DATA SCIENCE

for a fixed term 5-year appointment beginning as soon as possible, but no later than August 2022.

The duties of the present university lecturer will include providing teaching in accordance with the degree requirements, supervising and examining theses and dissertations, conducting research that is relevant for understanding the causes and consequences of changing family dynamics, and coordinating the activities of the Family Formation in Flux research team that is based at the University of Helsinki (https://fluxconsortium.fi/). In addition to the Flux project, the successful candidate will be affiliated with the Centre for Social Data Science (https://www.helsinki.fi/en/networks/centre-social-data-science) and with the Population Research Unit (https://www2.helsinki.fi/en/researchgroups/population-research).

QUALIFICATIONS

The appointee shall hold an applicable doctoral degree, for example in demography, sociology, economics, data science, or statistics, and the ability to provide high-quality teaching based on research and to supervise theses and dissertations. The degree requirement must be met by the end of the application deadline.

To successfully attend to the duties of the position, the appointee must have good English skills.

When assessing the qualifications of each applicant, attention will be paid to:

In the evaluation, special emphasis will be put on (1) research and publication track record in the field of demography, and (2) the ability to teach advanced quantitative methods in the social sciences. While part of the work of the successful candidate will include coordinating the activities of the Family Formation in Flux -project, and while we expect research contributions that are relevant for understanding changing family dynamics, at the application and selection stage we value equally existing research track records in social, health, and family demography.

WHAT WE OFFER

We are an equal opportunity employer and offer an attractive and diverse workplace in an inspiring environment. The annual gross salary range will be approx. 44,00065,500, depending on the appointees qualifications and experience. In addition, University of Helsinki offers comprehensive benefits to its employees, including occupational health care, opportunities for professional development, support for applying for research project funding as well as library, wellbeing and fitness services. Further information is available at https://www.helsinki.fi/en/about-us/careers. The employment contract will include a probationary period of six months.

The chosen applicant is expected to reside in Finland while employed by the University of Helsinki. The Faculty of Social Sciences provides assistance in relocation.

HOW TO APPLY

Please submit your application, together with the required attachments, through the University of Helsinki Recruitment System via the link Apply for the position. Applicants who are currently employed by the University of Helsinki are requested to submit their application via the SAP Fiori portal at https://msap.helsinki.fi. The closing date for applications is March 6, 2022.

Applicants are requested to enclose with their applications the following documents in English:

Detailed information on the CV and list of publications as well as the presentation of teaching skills can be obtained online: http://www.helsinki.fi/en/faculty-social-sciences/faculty/work-faculty-social-s....

The enclosures must be submitted as a SINGLE PDF file named as follows: fieldofposition_lastname_firstname. Other enclosures, such as recommendation letters or certificates, are not required.

FURTHER INFORMATION

Further information on the position and the work environment may be obtained from Professor Mikko Myrskyl, mikko.myrskyla@helsinki.fi.

Further information on the recruitment process may be obtained from HR Specialist Minna Maunula, minna.maunula@helsinki.fi.

Technical support for the recruitment system: rekrytointi@helsinki.fi.

Due date

06.03.2022 23:59 EET

Read more here:

University Lecturer in Demography and Data Science job with UNIVERSITY OF HELSINKI | 278749 - Times Higher Education (THE)

New Delhi: The Indian telecom industry has been at the heart of the technology driven disruption over the last decade. With over 986 million active wireless subscribers, the industry has an opportunity to also drive transformation for businesses across industries.

IANS spoke to Dr Sanjeev Chaube, EVP and Head, Big Data & Advanced Analytics at Vodafone Idea to get his insights on how Cloud has been at the core of their digital transformation and how they are leveraging intelligent technologies such as artificial intelligence (AI), machine learning (ML), Internet of Things (IoT) and analytics.

Dr Chaube believes that there is a critical need for telecom service providers to transform from 'Telco to Techno' and technologies such as 5G, Artificial Intelligence, Big Data Analytics, Cloud Computing, IoT / IIoT / AIoT, and Robotic Process Automation (RPA) will enable this digital transformation.

Q: What are the key strategic pillars of digital transformation at Indian Telcos as you look to transform from a connectivity provider to an orchestrator of value-added solutions that meet the needs of digitally-savvy, connected customers?

A: I believe businesses are fast turning to digital solutions to empower remote workforces, provide customers with better services, and create an immersive experience with increased visibility, resilience & agility. Moreover the need for eliminating traditional, hardware driven, and large expensive physical operations has been at the core of Digital Transformation worldwide across industries.

We need to understand that any transformation starts with data. Hence, a company captures customer data efficiently and leverages it to make smarter decisions. Goal should be to streamline key customer touchpoints to increase spends & reduce efforts.

The employees need to be well informed with relevant data, data driven decisioning culture, access to learning & development platforms empowering better organisational transformation.

Businesses need to shift from IT departments to consolidated enterprise platforms that can absorb & integrate latest technologies instantly. Datasets, processes, web and apps should be shared across the company and the data needs to be integrated to provide a single source of truth by building appropriate Data Lake, Data Warehouses/Marts over Cloud for massive storage & processing capacities.

However, the choice of technology stack whether Open-Source or Proprietary depends primarily on the Value vs Cost proposition. Also, the need for scope, scale, speed, quality & ease form the basis for finalising the tech stack.

Ensuring near Real-time Customer 360 view in form of Business Intelligence Reports and Dashboard for leadership and at different levels of management ensure better & contextual decision making.

Therefore, in my opinion, the key pillars and ingredients for effective Intelligent Digital Transformation Program are a) Artificial Intelligence/Machine Learning & Data Science for Data Driven Decision Making, b) Customer Relevance & Centricity Business Processes, c) Engage & Empower Employees aligned around customer, d) Digital First Strategy with leanest possible technology stack, e) Bring Speed, Scale, Simplicity & Values with Platform, f) Enhance Personalization and g) Single Source of Truth & Data Engineering

Q: How are you using Big Data, Advanced Analytics, Artificial Intelligence/Machine learning & Data Science to understand and cater to the needs of your customers better?

A: Big Data and Advanced Analytics is extensively adopted to improve customers experiences and business performance. Artificial Intelligence & Data Science techniques especially Machine/Deep Learning algorithms are significantly improving some of the crucial areas/services including Customer Segmentation, Targeted marketing, Personalised offerings & recommendations, Churn Prediction, Product development, Predictive Analytics, Call Drop Analysis, reducing fraud, Price optimization, Network experience optimization, Location based services etc.

Over the next five years, rise in mobile-phone penetration and decline in data costs will add 500 million new Internet users in India, creating opportunities for new businesses. Legacy data sets were all structured in nature, however over the last few years, with the explosion of data, unstructured data is gaining equal or more importance in the market. With the advent of 5G technology, the available data from voice, video, social media, messaging, IOT and all new 5G use case deployments are going to increase multifold from hereon.

This Big Data generated by telco's has all the features like massive volume, variety, velocity & veracity. It therefore becomes very critical for us to first organise and systematically store the data emerging from multiple applications within Data Ocean, Datalake, Data Warehouse /Data Marts be it using Cloudera Hadoop Ecosystem or Cloud infrastructure.

Secondly, this Big Data is then used to perform Data Analytics for decision making at different organisational levels from operations and strategy making perspective. All 5 stages of Data Science and Advanced Analytics lifecycle namely Descriptive, Diagnostic, Predictive, Prescriptive and Pre-Emptive Analytics are performed to extract information and patterns.

Machine learning use cases have great potential & value such as assisting with Customer Acquisition, Retention, Digital Engagement, Anomaly Detection, Root Cause Analysis, Predictive Maintenance, Ticket Classification, SLA Assurance and building intelligent networks with features like Self-Healing, Dynamic Optimization & Automated Network Designs. However, in order to work effectively, they require specific computational, pipeline and support infrastructure as well to support massive data & their parallel processing.

Predictive models whether Real-time or Batch processing basis business requirements leveraging structured data feeds such as demographic, usage, billing etc. or Unstructured data feeds like Chat, Text, Images, Video feeds are built to support business with insights beforehand.

Social media analytics is performed using Computer Vision, Natural Language Processing (NLP) & Text parsing techniques etc. to extract sentiments of customers across the country for decision making. Different Deep Learning Architectures of Neural networks ranging from Convolutional Neural Network, Recurrent Neural Networks, Self-Organizing Maps, Auto encoders, etc. are used appropriately as per desired intent.

Q: What did cloud technology allow you to do that you couldn't do before?

A: Cloud is a key enabler of Large-Scale Transformation: be it benefitting Consumer Analytics, Digital Adoption, and Network Monitoring & Management and bringing efficiency in Operations by facilitating faster decisions at scale and speed.

As on date, we are India's first Telco to build a full-fledged Big Data AI/ML Cloud Advanced Analytics Platform on AWS Cloud.

Before embarking on our cloud journey, our data sets were collected in silos across the country. Then, they were aggregated to run the various machine learning & deep learning algorithms.

Also, as the data sets were huge in volume, the available compute & GPU support for data consolidation in the legacy infrastructure was a challenge. All of these complexities added delays to our operations & hampered the productivity of the teams. With the new data science platform deployed over AWS, we are able to address all of these challenges & achieve much more.

Using the compute & memory power available on cloud, we can consolidate data & effectively run our machine / deep learning algorithms & also do parallel data processing with ease. The time to run these algorithms have already reduced by over 60 per cent, and with further optimization, we expect our data science platform to deliver more in the near future.

Additionally, availability of deep learning frameworks like Tensor flow, MXNet, Keras, Pytorch, Gluon, etc. along with pre-trained ML models as APIs help us experiment more with data sets, at scale & speed & eventually in turn improve our customer experience. This feature rich data science platform with its scalability, elasticity & pay-as-you-go commercial model helps consolidate operations & effectively manage costs.

In the future, we'll be able to integrate & effectively use this platform to address our challenges across new technology initiatives including 5G, Edge, IoT monetization, customer 360, Network Function Virtualization transition & so on.

Q: What are your views on the rapid worldwide progress in AIoT & IIoT Analytics for Connected Intelligence with Cloud Technologies?A: As per Fortune Business Insights, global IoT market size is projected to grow from USD 381.30 billion in 2021 to USD 1854.76 billion in 2028. While if we follow Researchandmarkets report, globally the AIoT market itself will reach $78.3 Billion by 2026. Also, the Global IIoT market which was valued at about $216.13 in 2020 and is expected to grow to about $1.1 trillion by 2028.

The above data clearly sets the roadmap and vision for telecom as a connectivity service provider not only for India but telco's globally.

The use of AI for decision making in IoT and data analytics will be crucial for efficient and effective smart city solutions in terms of decision making. This is in line with the Indian government's plan to develop 100 smart city projects including smart airports, smart railway stations, smart public healthcare services etc., where the 5G, AI, IoT & Cloud technologies will play a vital role in developing these cities & services.

The convergence of AI and Internet of Things (IoT) technologies and solutions (AIoT) is leading to "thinking" networks and systems. The AI-enabled edge device market will be the fastest-growing segment within the AIoT. AIoT automates data processing systems, converting raw IoT data into useful information.

AIoT solutions are the precursor to next-generation AI Decision as a Service (AIDaaS). With AIoT, AI is embedded into infrastructure components, such as programs, chipsets and edge computing, all interconnected with IoT networks. APIs are then used to extend interoperability between components at the device level, software level and platform level. These units will focus primarily on optimising system and network operations as well as extracting value from data.

These solutions will provide customer's ability to acquire leading wireless enabled AIoT products and reduce their time to market.

The key areas of AIoT solutions include Data Services, Predictive Asset Management, Intelligent Immersive Experience, Process Improvement, Next-Gen UI & UX and Intelligent Industrial Automation.

This Intelligent connectivity, Big Data & AI/ML technologies are expected to play a major role in key industrial areas such as Agricultural Productivity, Automotive, Transportation & logistics, Industrial & manufacturing operations, Healthcare, Public safety, and security.

Leveraged with AWS IoT Cloud services to collect, organise and analyse data from industrial & publicly installed sensors & devices at scale and for near real-time decision making, AIoT enables industries to seamlessly manage and control actions across the edge and cloud.

The key components for Artificial Intelligence + IoT (AIoT) Analytics consist of Sensors, IoT Gateway, Network, Storage & Analytics Engine for Data Analysis, Monitoring and Intelligent Automation for actions & notifications at the Edge.

Artificial Intelligence & Machine learning algorithms are leverage both at Cloud & Edge over emerging data from IoT sensors such as Visual/Optical Sensors, Temperature Sensors, Pressure Sensors, Infrared Sensors, Gas, Gyroscope Sensors, Accelerometers, Humidity Sensors, Level Sensors etc. to derive descriptive & predictive insights for effective & real-time monitoring, notifications and action fulfilment.

Continued here:

We are India's 1st telco to build Big Data AI/ML Cloud Advanced Analytics Platform on AWS: Vodafone Idea - ETTelecom.com