Cloud Hosting

By Jeff Hollan

Jeff Hollan, Director of Product Management, Snowflake, highlights the anticipated developments in 2024 as artificial intelligence becomes integrated into the business operations

Data engineering will evolve and be highly valued in an AI world.

Theres been a lot of chatter that the AI revolution will replace the role of data engineers. Thats not the case, and in fact their data expertise will be more critical than ever just in new and different ways. To keep up with the evolving landscape, data engineers will need to understand how generative AI adds value. The data pipelines built and managed by data engineers will be perhaps the first place to connect with large language models for organizations to unlock value. Data engineers will be the ones who understand how to consume a model and plug it into a data pipeline to automate the extraction of value. They will also be expected to oversee and understand the AI work.

Data scientists will have more fun.

Just as cloud infrastructure forced IT organizations to learn new skill sets by moving from builders of infrastructure and software, to managers of third-party infrastructure and software vendors, data science leaders will have to learn to work with external vendors. It will be an increasingly important skill to be able to pick the right vendors of AI models to engage with, similar to how data scientists today choose which frameworks to use for specific use cases. The data scientist of tomorrow might be responsible for identifying the right vendors of AI models to engage with, determining how to feed the right context into a large language model (LLM), minimizing hallucinations, or prompting LLMs to answer questions correctly through context and formalizing metadata. These are all new and exciting challenges that will keep data scientists engaged and hopefully inspire the next generation to get into the profession.

BI analysts will have to uplevel.

Today, business intelligence analysts generally create and present canned reports. When executives have follow-up questions, the analysts then have to run a new query to generate a supplemental report. In the coming year, executives will expect to interact directly with data summarized in that overview report using natural language. This self-service will free up analysts to work on deeper questions, bringing their own expertise to what the organization really should be analyzing, and ultimately upleveling their role to solve some of the challenges AI cant.

(The author is Jeff Hollan, Director of Product Management, Snowflake, and the views expressed ins this article are his own)

Follow this link:

The Future of Data Engineering in an AI-Driven Landscape - CXOToday.com