Cloud Hosting

Explore solutions to poor data quality challenges in investigations with insights from Tom Saltsberg, Product Manager at Cognyte. Discover how Decision Intelligence enhances analytical outcomes.

Investigators and intelligence analysts are frequently challenged to make sense of poor quality. Poor data quality presents fundamentally unique challenges in part because poor is a relative term and in most cases, analysts cannot recognize whether data is of poor quality at first glance, thereby needing to assess it beforehand, which is time-consuming and inefficient.

Poor data quality can manifest in many ways and often be attributed to human error. This could include data with incorrect information, including misspellings and typos. Alternatively, it could manifest as partial or missing data or data inadvertently duplicated or ingested incorrectly. Data thats correct and otherwise high quality but considered outdated could also contribute to poor data quality, depending on the investigations objectives. Investigators must also grapple with fraudulent data data that are not credible and deliberately misleading a virulent strain of poor-quality data intended to distract and confuse them. Ironically, fraudulent data will probably seem the most credible since the actors who prepared it invested time and effort to give it a sense of authenticity.

There are many techniques for cleaning and validating data. Cross-referencing data with other sources is crucial for validating data quality and is easily achievable with small data sets. But large data sets can make it difficult or in some cases, impossible for analysts to correlate this data manually.

One of the main goals for investigations assessing large volumes of data (customs, finance, etc.) is to identify trends and anomalies, such as undervaluations or misreports, amid the deluge via quantitative assessment. In law enforcement investigations, on the other hand, data sets could be considerably smaller and easily organized, with fewer sources to be correlated in a more qualitative assessment.

See More: How to Integrate New Data During Acquisition

What happens when only a single source of data and cross-referencing is impossible? This is an obstacle for analysts correlating data manually and machine learning (ML) algorithms assessing data in volume for statistical insights. Theres a misconception that big data can answer these challenges for law enforcement agencies, but big data on its own is a solution.

Multiple approaches could be used to overcome these data quality challenges, including processing the data on a mass scale using data science and ML. However, data science requires that we understand and carefully deliberate on the features to be calculated by the ML model at the outset of the solution design. Data quality plays a major role in this feature-generation phase when techniques are optimized to understand and complete missing or low-quality data values.

Data quality challenges can impact various applications, investigative and otherwise. Customs fraud is a classic example, and its rampant and exceedingly difficult to track. Customs documentation is often scanned, making extracting and referencing relevant data harder. These documents could also include ad hoc manual notations from port/customs administrators (human error may be introduced here), and language gaps can also be problematic in customs documentation.

Numerous examples of customs fraud perpetrated or aided by leveraging intentionally misleading (poor quality) data. This could include shipped goods deliberately misclassified or undervalued for tax evasion purposes. Also, information intended to mislead the authorities could manifest as partial or obfuscated, fragmented data obscuring certain trade activities true purpose. For instance, vehicle parts are shipped separately, as assembled vehicles, to avoid regulation constraints or taxation (a technique known as structuring).

Law enforcement is another domain thats rife with data quality issues and these issues, in some cases, arent identified until decades after the crimes were investigated and justice meted out by the legal system. High-profile justice campaigns are being mounted, consolidated, and visualized based on flawed forensics data or human error in the case files.

According to the Innocence ProjectOpens a new window , false or misleading forensic evidence has contributed to 24% of wrongful convictions nationally in the U.S., based on data from the National Registry of Exonerations, which tracks DNA and non-DNA-based exonerations. This includes convictions based on forensic evidence that is unreliable or invalid.

Disinformation (fake news) is a fascinating example of poor data quality. Though the data quality of disinformation is fundamentally bad, considerable care has been taken in many cases to make the data appear good.

Disinformation strategies are often applied in geopolitics and organized crime. They could include decoy and fake data sets designed to interrupt or influence democratic elections or distort intelligence gathering, for example. Disinformation is also unique in that it can be practiced enormously, including content planted in mass media and social media to sway public opinion and votes.

Poor data quality can also amplify known analyst biasesOpens a new window . Cognitive bias is a foundational problem affecting criminal investigation in all its forms, and its exceedingly difficult to compensate for with existing technological methods.

Examples could include anchoring bias (analyst over-reliance on primary information), groupthink bias, and bias that artificially inflates the importance of information that supports the analysts position. In the future, AI technology advancements will hopefully make it possible to reduce or eliminate these biases so that clear thinking can prevail.

See More: Why Composable Data Matters for Collaboration and Safety

Decision intelligence platforms that provide scalable, multi-featured data collection, fusion, and analysis capabilities are naturally beneficial for completing the investigative picture when data is incomplete or inaccurate.

A centralized system is beneficial it allows analysts to fuse a lot of data from multiple and siloed data sources agnostic to the format (e.g., this overcomes the challenges with scanned documents used to perpetuate customs fraud). Data fusion also enables consolidating and visualizing the data for more efficient assessment by creating entities and profiles and not inspecting every data point independently.

A centralized decision intelligence system also enables data enrichment from multiple angles to provide a more holistic view of the data and the insights reflected therein. Finally, a centralized system automates cross-referencing, overcoming many of the above challenges. The data ingested into a centralized system can be readily cross-referenced, verified, and validated to improve overall data quality.

Centralized decision intelligence ultimately enables analysts to derive more meaningful insights from data whether the data is high quality or low quality to help them make more informed decisions and prioritize effectively during investigations.

How can decision intelligence transform investigations? Let us know on FacebookOpens a new window , XOpens a new window , and LinkedInOpens a new window . Wed love to hear from you!

Image Source: Shutterstock

See the original post here:

Mitigating Poor Data Quality With Decision Intelligence Solutions - Spiceworks News and Insights