Solutions Review Executive Editor Tim King curated this list of notable analytics and data science news for the week of May 3, 2024.

Keeping tabs on all the most relevant analytics and data science news can be a time-consuming task. As a result, our editorial team aims to provide a summary of the top headlines from the last week, in this space. Solutions Review editors will curate vendor product news, mergers and acquisitions, venture capital funding, talent acquisition, and other noteworthy analytics and data science news items.

FlightStream can capture subsonic to supersonic flows, including compressible effects and a unique surface vorticity capability. It leverages the strengths of panel method flow solvers and enhances them with modern computational techniques to provide a fast solver capable of handling complex aerodynamic phenomena.

Read on for more

By supporting any JDK distribution including those from Azul, Oracle, Amazon, Eclipse, Microsoft, Red Hat and others, Azul Intelligence Cloud delivers key benefits across an enterprises entire Java fleet.

Read on for more

This authorization builds onAzure Databricks FedRAMP High and IL5 authorizations and further demonstrates Databricks commitment to meeting the federal governments requirements for handling highly sensitive and mission-critical defense Controlled Unclassified Information (CUI) across a wide variety of data analytics and AI use cases.

Read on for more

In the last 12 months, DataRobot introducedindustry-first generative AI functionality, launched theDataRobot Generative AI Catalyst Program to jumpstart high-priority use cases, and announced expanded collaborations with NVIDIAandGoogle Cloudto supercharge AI solutions with world-class performance and security.

Read on for more

The platform enables data scientists and CISO teams to gain valuable understanding and insights into AI systems risks and challenges, alongside comprehensive protection and alerts. DeepKeep is already deployed by leading global enterprises in the finance, security, and AI computing sectors.

Read on for more

MicroStrategy AI, introduced in 2023, is now in its third GA release, which includes enhanced AI explainability, automated workflows, and several other features designed to increase convenience, reliability, and flexibility for customers. The company also launched MicroStrategyAuto Express, allowing anyone to create and share AI-powered bots and dashboards free for 30 days.

Read on for more

The Predactiv platform emerged from the success of ShareThis, a leading data and programmatic advertising provider, recognized for engineering expertise and pioneering the use of AI in audience and insight creation.

Read on for more

Salesforce Inc. rolled out data visualization and AI infrastructure improvements for its Tableau Software platform today that increase the usability of its product for data analysts and expand its scalability for artificial intelligence.

Read on for more

Sigma is building the AI Toolkit for Business, which will bring powerful AI and ML innovations from the data ecosystem into an intuitive interface that anyone can use. With forms and actions, a customers data app is constantly up-to-date because its directly connected to the data in their warehouse.

Read on for more

Watch this space each week as our editors will share upcoming events, new thought leadership, and the best resources from Insight Jam, Solutions Reviews enterprise tech community for business software pros. The goal? To help you gain a forward-thinking analysis and remain on-trend through expert advice, best practices, predictions, and vendor-neutral software evaluation tools.

With the next Solutions Spotlight event, the team at Solutions Review has partnered with leading developer tools provider Infragistics. In this presentation, were bringing two of the biggest trends in the market today low-code app development and embedded BI.

Register free on LinkedIn

With the next Expert Roundtable event, the team at Solutions Review has partnered with Databricks and ZoomInfo to cover why bad data is the problem, how Databricks and ZoomInfo help companies build a unified data foundation that fixes the bad data problem, and how this foundation can be leveraged to easily scale and use data + AI for GenAI.

Register free on LinkedIn

This summit is designed for leaders looking to modernize their organizations data integration capabilities across analytics, operations and artificial intelligence use cases. Dive deep into the core use cases of data integration as it pertains to analytics, operations, and artificial intelligence. Learn how integrating data can drive operational excellence, enhance analytical capabilities, and fuel AI innovations.

Register free

Getting data from diverse data producers to data consumers to meet business needs is a complicated and time-consuming task that often traverses many products. The incoming data elements are enriched, correlated, and integrated so that the consumption-ready data products are meaningful, timely and trustworthy.

Read on for more

Debbie describes herself as a very social person and enjoys working with people., she cares about any problems colleagues experience and has a natural leaning towards wanting to help. On top of that, Debbie is also a musician, which lends itself to wanting order, structure, systems, and rules. This all adds up to make Debbie very good at her most recent role of Data Governance Manager for Solent University.

Read on for more

For consideration in future analytics and data science news roundups, send your announcements to the editor: tking@solutionsreview.com.

Read more from the original source:

Analytics and Data Science News for the Week of May 3; Updates from Databricks, DataRobot, MicroStrategy & More - Solutions Review

Picture by Snapwire, on Pexels

This blog post is an updated version of part of a conference talk I gave on GOTO Amsterdam last year. The talk is also available to watch online.

As a Machine Learning Product Manager, I am fascinated by the intersection of Machine Learning and Product Management, particularly when it comes to creating solutions that provide value and positive impact on the product, company, and users. However, managing to provide this value and positive impact is not an easy job. One of the main reasons for this complexity is the fact that, in Machine Learning initiatives developed for digital products, two sources of uncertainty intersect.

From a Product Management perspective, the field is uncertain by definition. It is hard to know the impact a solution will have on the product, how users will react to it, and if it will improve product and business metrics or not Having to work with this uncertainty is what makes Product Managers potentially different from other roles like Project Managers or Product Owners. Product strategy, product discovery, sizing of opportunities, prioritization, agile, and fast experimentation, are some strategies to overcome this uncertainty.

The field of Machine Learning also has a strong link to uncertainty. I always like to say With predictive models, the goal is to predict things you dont know are predictable. This translates into projects that are hard to scope and manage, not being able to commit beforehand to a quality deliverable (good model performance), and many initiatives staying forever as offline POCs. Defining well the problem to solve, initial data analysis and exploration, starting small, and being close to the product and business, are actions that can help tackle the ML uncertainty in projects.

Mitigating this uncertainty risk from the beginning is key to developing initiatives that end up providing value to the product, company, and users. In this blog post, Ill deep-dive into my top 3 lessons learned when starting ML Product initiatives to manage this uncertainty from the beginning. These learnings are mainly based on my experience, first as a Data Scientist and now as an ML Product Manager, and are helpful to improve the likelihood that an ML solution will reach production and achieve a positive impact. Get ready to explore:

I have to admit, I have learned this the hard way. Ive been involved in projects where, once the model was developed and prediction performance was determined to be good enough, the models predictions werent really usable for any specific use case, or were not useful to help solve any problem.

There are many reasons this can happen, but the ones Ive found more frequently are:

To start an ML initiative on the right foot, it is key to start with the good problem to solve. This is foundational in Product Management, and recurrently reinforced product leaders like Marty Cagan and Melissa Perri. It includes product discovery (through user interviews, market research, data analysis), and sizing and prioritization of opportunities (by taking into account quantitative and qualitative data).

Once opportunities are identified, the second step is to explore potential solutions for the problem, which should include Machine Learning and GenAI techniques, if they can help solve the problem.

If it is decided to try out a solution that includes the use of predictive models, the third step would be to do an end-to-end definition and design of the solution or system. This way, we can ensure the requirements on how to use the predictions by the system, influence the design and implementation of the predictive piece (what to predict, data to be used, real-time vs batch, technical feasibility checks).

However, Id like to add there might be a notable exception in this topic. Starting from GenAI solutions, instead of from the problem, can make sense if this technology ends up truly revolutionizing your sector or the world as we know it. There are a lot of discussions about this, but Id say it is not clear yet whether that will happen or not. Up until now, we have seen this revolution in very specific sectors (customer support, marketing, design) and related to peoples efficiency when performing certain tasks (coding, writing, creating). For most companies though, unless its considered R&D work, delivering short/mid-term value still should mean focusing on problems, and considering GenAI just as any other potential solution to them.

Tough experiences lead to this learning as well. Those experiences had in common a big ML project defined in a waterfall manner. The kind of project that is set to take 6 months, and follow the ML lifecycle phase by phase.

What could go wrong, right? Let me remind you of my previous quote With predictive models, the goal is to predict things you dont know are predictable! In a situation like this, it can happen that you arrive at month 5 of the project, and during the model evaluation realize there is no way the model is able to predict whatever it needs to predict with good enough quality. Or worse, you arrive at month 6, with a super model deployed in production, and realize it is not bringing any value.

This risk combines with the uncertainties related to Product, and makes it mandatory to avoid big, waterfall initiatives if possible. This is not something new or related only to ML initiatives, so there is a lot we can learn from traditional software development, Agile, Lean, and other methodologies and mindsets. By starting small, validating assumptions soon and continuously, and iteratively experimenting and scaling, we can effectively mitigate this risk, adapt to insights and be more cost-efficient.

While these principles are well-established in traditional software and product development, their application to ML initiatives is a bit more complex, as it is not easy to define small for an ML model and deployment. There are some approaches, though, that can help start small in ML initiatives.

Rule-based approaches, simplifying a predictive model through a decision tree. This way, predictions can be easily implemented as if-else statements in production as part of the functionality or system, without the need to deploy a model.

Proofs of Concept (POCs), as a way to validate offline the predictive feasibility of the ML solution, and hint on the potential (or not) of the predictive step once in production.

Minimum Viable Products (MVPs), to first focus on essential features, functionalities, or user segments, and expand the solution only if the value has been proven. For an ML model this can mean, for example, only the most straightforward, priority input features, or predicting only for a segment of data points.

Buy instead of build, to leverage existing ML solutions or platforms to help reduce development time and initial costs. Only when proved valuable and costs increase too much, might be the right time to decide to develop the ML solution in-house.

Using GenAI as an MVP, for some use cases (especially if they involve text or images), genAI APIs can be used as a first approach to solve the prediction step of the system. Tasks like classifying text, sentiment analysis, or image detection, where GenAI models deliver impressive results. When the value is validated and if costs increase too much, the team can decide to build a specific traditional ML model in-house.

Note that using GenAI models for image or text classification, while possible and fast, means using a way too big an complex model (expensive, lack of control, hallucinations) for something that could be predicted with a much simpler and controllable one. A fun analogy would be the idea of delivering a pizza with a truck: it is feasible, but why not just use a bike?

Data is THE recurring problem Data Scientist and ML teams encounter when starting ML initiatives. How many times have you been surprised by data with duplicates, errors, missing batches, weird values And how different that is from the toy datasets you find in online courses!

It can also happen that the data you need is simply not there: the tracking of the specific event was never implemented, collection and proper ETLs where implemented recently I have experienced how this translates into having to wait some months to be able to start a project with enough historic and volume data.

All this relates to the adage Garbage in, garbage out: ML models are only as good as the data theyre trained on. Many times, solutions have a bigger potential to be improve by improving the data than by improving the models (Data Centric AI). Data needs to be sufficient in volume, historic (data generated during years can bring more value than the same volume generated in just a week), and quality. To achieve that, mature data governance, collection, cleaning, and preprocessing are critical.

From the ethical AI point of view, data is also a primary source of bias and discrimination, so acknowledging that and taking action to mitigate these risks is paramount. Considering data governance principles, privacy and regulatory compliance (e.g. EUs GDPR), is also key to ensure a responsible use of data (especially when dealing with personal data).

With GenAI models this is pivoting: huge volumes of data are already used to train them. When using these types of models, we might not need volume and quality data for training, but we might need it for fine-tuning (see Good Data = Good GenAI), or to construct the prompts (nurture the context, few-shot learning, Retrieval Augmented Generation I explained all these concepts in a previous post!).

It is important to note that by using these models we are losing control of the data used to train them, and we can suffer from the lack of quality or type of data used there: there are many known examples of bias and discrimination in GenAI outputs that can negatively impact our solution. A good example was Bloombergs article on how How ChatGPT is a recruiters dream tool tests show theres racial bias. LLM leaderboards testing for biases, or LLMs specifically trained to avoid these biases can be useful in this sense.

We started this blogpost discussing what makes ML Product initiatives especially tricky: the combination of the uncertainty related to developing solutions in digital products, with the uncertainty related to trying to predict things through the use of ML models.

It is comforting to know there are actionable steps and strategies available to mitigate these risks. Yet, perhaps the best ones, are related to starting the initiatives off on the right foot! To do so, it can really help to start with the right problem and an end-to-end design of the solution, reduce initial scope, and prioritize data quality, volume, and historical accuracy.

I hope this post was useful and that it will help you challenge how you start working in future new initiatives related to ML Products!

More:

Starting ML Product Initiatives on the Right Foot - Towards Data Science

Since its publication in a 2017 paper by OpenAI, Proximal Policy Optimization (PPO) is widely regarded as one of the state-of-the-art algorithms in Reinforcement Learning. Indeed, PPO has demonstrated remarkable performances across various tasks, from attaining superhuman performances in Dota 2 teams to solving a Rubiks cube with a single robotic hand while maintaining three main advantages: simplicity, stability, and sample efficiency.

However, implementing RL algorithms from scratch is notoriously difficult and error-prone, given the numerous error sources and implementation details to be aware of.

In this article, well focus on breaking down the clever tricks and programming concepts used in a popular implementation of PPO in JAX. Specifically, well focus on the implementation featured in the PureJaxRL library, developed by Chris Lu.

Disclaimer: Rather than diving too deep into theory, this article covers the practical implementation details and (numerous) tricks used in popular versions of PPO. Should you require any reminders about PPOs theory, please refer to the references section at the end of this article. Additionally, all the code (minus the added comments) is copied directly from PureJaxRL for pedagogical purposes.

Proximal Policy Optimization is categorized within the policy gradient family of algorithms, a subset of which includes actor-critic methods. The designation actor-critic reflects the dual components of the model:

Additionally, this implementation pays particular attention to weight initialization in dense layers. Indeed, all dense layers are initialized by orthogonal matrices with specific coefficients. This initialization strategy has been shown to preserve the gradient norms (i.e. scale) during forward passes and backpropagation, leading to smoother convergence and limiting the risks of vanishing or exploding gradients[1].

Orthogonal initialization is used in conjunction with specific scaling coefficients:

The training loop is divided into 3 main blocks that share similar coding patterns, taking advantage of Jaxs functionalities:

Before going through each block in detail, heres a quick reminder about the jax.lax.scan function that will show up multiple times throughout the code:

A common programming pattern in JAX consists of defining a function that acts on a single sample and using jax.lax.scan to iteratively apply it to elements of a sequence or an array, while carrying along some state. For instance, well apply it to the step function to step our environment N consecutive times while carrying the new state of the environment through each iteration.

In pure Python, we could proceed as follows:

However, we avoid writing such loops in JAX for performance reasons (as pure Python loops are incompatible with JIT compilation). The alternative is jax.lax.scan which is equivalent to:

Using jax.lax.scan is more efficient than a Python loop because it allows the transformation to be optimized and executed as a single compiled operation rather than interpreting each loop iteration at runtime.

We can see that the scan function takes multiple arguments:

Additionally, scan returns:

Finally, scan can be used in combination with vmap to scan a function over multiple dimensions in parallel. As well see in the next section, this allows us to interact with several environments in parallel to collect trajectories rapidly.

As mentioned in the previous section, the trajectory collection block consists of a step function scanned across N iterations. This step function successively:

Scanning this function returns the latest runner_state and traj_batch, an array of transition tuples. In practice, transitions are collected from multiple environments in parallel for efficiency as indicated by the use of jax.vmap(env.step, )(for more details about vectorized environments and vmap, refer to my previous article).

After collecting trajectories, we need to compute the advantage function, a crucial component of PPOs loss function. The advantage function measures how much better a specific action is compared to the average action in a given state:

Where Gt is the return at time t and V(St) is the value of state s at time t.

As the return is generally unknown, we have to approximate the advantage function. A popular solution is generalized advantage estimation[3], defined as follows:

With the discount factor, a parameter that controls the trade-off between bias and variance in the estimate, and t the temporal difference error at time t:

As we can see, the value of the GAE at time t depends on the GAE at future timesteps. Therefore, we compute it backward, starting from the end of a trajectory. For example, for a trajectory of 3 transitions, we would have:

Which is equivalent to the following recursive form:

Once again, we use jax.lax.scan on the trajectory batch (this time in reverse order) to iteratively compute the GAE.

Note that the function returns advantages + traj_batch.value as a second output, which is equivalent to the return according to the first equation of this section.

The final block of the training loop defines the loss function, computes its gradient, and performs gradient descent on minibatches. Similarly to previous sections, the update step is an arrangement of several functions in a hierarchical order:

Lets break them down one by one, starting from the innermost function of the update step.

This function aims to define and compute the PPO loss, originally defined as:

Where:

However, the PureJaxRL implementation features some tricks and differences compared to the original PPO paper[4]:

Heres the complete loss function:

The update_minibatch function is essentially a wrapper around loss_fn used to compute its gradient over the trajectory batch and update the model parameters stored in train_state.

Finally, update_epoch wraps update_minibatch and applies it on minibatches. Once again, jax.lax.scan is used to apply the update function on all minibatches iteratively.

From there, we can wrap all of the previous functions in an update_step function and use scan one last time for N steps to complete the training loop.

A global view of the training loop would look like this:

We can now run a fully compiled training loop using jax.jit(train(rng)) or even train multiple agents in parallel using jax.vmap(train(rng)).

There we have it! We covered the essential building blocks of the PPO training loop as well as common programming patterns in JAX.

To go further, I highly recommend reading the full training script in detail and running example notebooks on the PureJaxRL repository.

Thank you very much for your support, until next time

Full training script, PureJaxRL, Chris Lu, 2023

[1] Explaining and illustrating orthogonal initialization for recurrent neural networks, Smerity, 2016

[2] Initializing neural networks, DeepLearning.ai

[3] Generalized Advantage Estimation in Reinforcement Learning, Siwei Causevic, Towards Data Science, 2023

[4] Proximal Policy Optimization Algorithms, Schulman et Al., OpenAI, 2017

See more here:

Breaking down State-of-the-Art PPO Implementations in JAX - Towards Data Science

Deloitte has acquired Gryphon Scientific for an undisclosed sum as part of efforts to broaden its biosafety, biosecurity, public health and emergency preparedness and response capabilities.

Through the transaction, Deloitte said Monday it will absorb Gryphons scientists, programmers, data analysts, public health specialists and planning and policy professionals with experience in scientific communications, modeling, data science and risk assessment.

The two companies will also develop artificial intelligence applications through Deloittes Federal Health AI Accelerator to improve public health and safety and help customers prepare for chemical threats, biothreats and other biological emergencies.

Beth Meagher, U.S. federal health sector leader and principal at Deloitte Consulting, said the addition of Gryphons professionals and leadership reflects an enhancement to the consulting firms data analytics and advanced tech capabilities.

Our federal health practice is excited to lead the way for U.S. government and public services (GPS) to push the boundaries and bring our clients to the forefront of AI-enabled, mission-driven work, Meagher noted. This enhances the types of data-driven technology and scientific experience that we can offer to federal agencies and strengthens our ability to support government leaders in their efforts to safeguard the security of our nation and the health and safety of our people.

See original here:

Deloitte Broadens Biosecurity, Public Health Capabilities With Gryphon Scientific Acquisition; Beth Meagher Quoted - GovCon Wire

Image by Author

I once spoke with a guy who bragged that, armed only with some free LinkedIn courses and an outdated college Intro to SQL course, hed managed to bag a six-figure job in data science. Nowadays, most people struggling to get a good data science job will agree thats unlikely to happen. Does that mean the data science job category is a popped bubble or worse, that it hasnt yet burst, but is about to?

In short, no. Whats happened is that data science used to be an undersaturated field, easy to get into if you used the right keywords on your resume. Nowadays, employers are a little more discerning and often have specific skill sets in mind that theyre looking for.

The bootcamps, free courses, and Hello World projects dont cut it anymore. You need to prove specific expertise and nail your data science interview, not just drop buzzwords. Not only that, but the shine of data scientist has worn off a little. For a long time, it was the sexiest job out there. Now? Other fields, like AI and machine learning, are just a bit sexier.

That all being said, there are still more openings in data science than there are applicants, and reliable indicators say the field is growing, not shrinking.

Not convinced? Lets look at the data.

Over the course of this article, Ill drill down into multiple graphs, charts, figures, and percentages. But lets start with just one percentage from one outstandingly reputable source: The Bureau of Labor Statistics.

The BLS predicts that there will be a 35 percent change in employment from 2022 to 2032 for data scientists. In short, in 2032, there will be about a third more jobs in data science than there were in 2022. For comparison, the average growth rate for all jobs is 3 percent. Keep that number in mind as you go through the rest of this article.

The BLS does not think that data science is a bubble waiting to burst.

Now we can start getting into a bit of the nitty gritty. The first signs people point to as signs of a popped or impending bubble pop are the mass layoffs in data science.

Its true that the numbers dont look good. Starting in 2022 and continuing through 2024, the tech sector in general experienced 430k layoffs. Its difficult to tease out data science-specific data from those numbers, but the best guesses are that around 30 percent of those were in data science and engineering.

Source: https://techcrunch.com/2024/04/05/tech-layoffs-2023-list/

However, thats not a burst bubble of data science. Its a little smaller in scope than that its a pandemic bubble popping. In 2020, as more people stayed home, profits rose, and money was cheap, FAANG and FAANG-adjacent companies scooped up record numbers of tech workers, only to lay many of them off just a few years later.

If you zoom out and look at the broader picture of hirings and layoffs, youll be able to see that the post-pandemic slump is a dip in an overall rising line, which is even now beginning to recover:

Source: https://www.statista.com/

You can clearly see the huge dip in tech layoffs during 2020 as the market tightened, and then the huge spike starting in Q1 of 2022 as layoffs began. Now, in 2024, the number of layoffs is smaller than in 2023.

Another scary stat often touted is that FAANG companies shuttered their job openings by 90% or more. Again, this is most in reaction to a widely high number of job openings during the pandemic.

That being said, job openings in the tech sector are still lower than they were pre-pandemic. Below, you can see an adjusted chart showing demand for tech jobs relative to February 2020. Its clear to see that the tech sector took a blow its not recovering from any time soon.

Source: https://www.hiringlab.org/2024/02/20/labor-market-update-tech-jobs-below-pre-pandemic-levels/

However, lets look a little closer at some real numbers. Looking at the chart below, while job openings are indubitably down from their 2022 peak, the overall number of openings is actually increasing up 32.4% from the lowest point.

Source: https://www.trueup.io/job-trend

If you look at any labor and news reports online, youll see theres a bit of an anti-remote, anti-tech backlash happening at the moment. Meta, Google, and other FAANG companies, spooked by the bargaining power that employees enjoyed during the pandemic heights, are now pushing for return-to-office mandates (data science jobs and other tech jobs are often remote) and laying off large quantities of employees somewhat unnecessarily, judging by their revenue and profit reports.

Just to give one example, Googles parent company Alphabet laid off over 12,000 employees over the course of 2023 despite growth across its ad, cloud, and services divisions.

This is just one facet with which to examine the data, but part of the reason companies are doing these layoffs is more to do with making the board happy rather than any decreased need for data scientists.

I find that people believing were in a data science bubble are most often those who dont really know what data scientists do. Think of that BLS stat and ask yourself: why does this well-informed government agency believe that theres strong growth in this sector?

Its because the need for data scientists cannot go away. While the names might be changed AI expert or ML Cloud Specialist rather than Data Scientist the skills and tasks that data scientists perform cant be outsourced, dropped, decreased, or automated.

For example, predictive models are essential for businesses to forecast sales, predict customer behavior, manage inventory, and anticipate market trends. This enables companies to make informed decisions, plan strategically for the future, and maintain competitive advantages.

In the financial sector, data science plays a crucial role in identifying suspicious activities, preventing fraud, and mitigating risks. Advanced algorithms analyze transaction patterns to detect anomalies that may indicate fraud, helping protect businesses and consumers alike.

NLP enables machines to understand and interpret human language, powering applications like chatbots, sentiment analysis, and language translation services. This is critical for improving customer service, analyzing social media sentiment, and facilitating global communication.

I could list dozens more examples demonstrating that data science is not a fad, and data scientists will always be in demand.

Revisiting my anecdote from earlier, part of the reason it feels like were in a bubble that is either popping or about to pop is the perception of data science as a career.

Back in 2011, Harvard Business Review famously called it the sexiest job of the decade. In the intervening years, companies hired more data scientists than they knew what to do with, often unsure about what data scientists actually did.

Now, a decade and a half later, the field is a little wiser. Employers understand that data science is a broad field, and are more interested in hiring machine learning specialists, data pipeline engineers, cloud engineers, statisticians, and other specialties that broadly fall under the data science hat but are more specialized.

This also helps explain why this idea of walking into six figure job straight out of a bachelor's degree used to be the case - since employers didn't know better - but now is impossible to do. The lack of easy data science jobs makes it feel like the market is tighter. It's not; data shows job openings are still high and demand is still greater than the graduates coming out with appropriate degrees. But employers are more discerning and unwilling to take a chance on untried college grads with no demonstrated experience.

Finally, you can take a look at the tasks that data scientists do and ask yourself what companies would do without those tasks getting done.

If you don't know much about data science, you might guess that companies can simply automate this work, or even go without. But if you know anything about the actual tasks data scientists do, you understand that the job is, currently, irreplaceable.

Think of how things were in the 2010s: that guy I talked about, with just a basic understanding of data tools, catapulted himself into a lucrative career. Things arent like that anymore, but this recalibration isnt a sign of a bursting bubble as some believe. Instead, its the field of data science maturing. The entry-level data science field may be oversaturated, but for those with specialized skills, deep knowledge, and practical experience, the field is wide open.

Furthermore, this narrative of a bubble is fueled by a misunderstanding of what a bubble actually represents. A bubble occurs when the value of something (in this case, a career sector) is driven by speculation rather than actual intrinsic worth. However, as we covered, the value proposition of data science is tangible and measurable. Companies need data scientists, plain and simple. Theres no speculation there.

Theres also a lot of media sensationalism surrounding the layoffs in big tech. While these layoffs are significant, they reflect broader market forces rather than a fundamental flaw in the data science discipline. Dont get caught up in the headlines.

Finally, its also worth noting that the perception of a bubble may stem from how data science itself is changing. As the field matures, the differentiation between roles becomes more pronounced. Job titles like data engineering, data analysis, business intelligence, machine learning engineering, and data science are more specific, and require a more niche skill set. This evolution can make the data science job market appear more volatile than it is, but in reality, companies just have a better understanding of their data science needs and can recruit for their specialities.

If you want a job in data science, go for it. Theres very little chance were actually in a bubble. The best thing you can do is, as Ive indicated, pick your specialty and develop your skills in that area. Data science is a broad field, spilling over into different industries, languages, job titles, responsibilities, and seniorities. Select a specialty, train the skills, prep for the interview, and secure the job.

Nate Rosidi is a data scientist and in product strategy. He's also an adjunct professor teaching analytics, and is the founder of StrataScratch, a platform helping data scientists prepare for their interviews with real interview questions from top companies. Nate writes on the latest trends in the career market, gives interview advice, shares data science projects, and covers everything SQL.

Read more here:

Is Data Science a Bubble Waiting to Burst? - KDnuggets