Copyright TechTarget - All rights reserved https://cyber.law.harvard.edu/rss/rss.html Techtarget Feed Generator en Sun, 14 Jun 2026 00:02:36 GMT https://www.techtarget.com/searchdatamanagement editor@techtarget.com <p>Effective data integration is essential for generating accurate analytics insights and AI outcomes in big data applications. But big data integration requires a shift away from traditional integration techniques to handle large volumes of diverse data often collected and processed at high velocity.</p> <p><a href="https://www.techtarget.com/searchdatamanagement/The-ultimate-guide-to-big-data-for-businesses">Big data environments</a> provide new opportunities to derive insights from unstructured and semistructured data, such as website and application logs, emails, social media posts, images and IoT data streams. Conventional integration approaches fall short when teams need to work with this data, said Rosaria Silipo, a data scientist, author and co-host of the "My Data Guest" podcast.</p> <p>Data integration is especially challenging when volume, variety and velocity -- the core 3 V's of big data -- are all factors in analytics and AI applications. Ad hoc integration for individual projects isn't viable in such scenarios, Silipo said. To avoid problems and <a href="https://www.techtarget.com/searchbusinessanalytics/feature/6-big-data-benefits-for-businesses">maximize business value</a>, data leaders must develop a comprehensive integration strategy that addresses big data's scale and complexity.</p> <section class="section main-article-chapter" data-menu-title="New challenges, new approaches"> <h2 class="section-title"><i class="icon" data-icon="1"></i>New challenges, new approaches</h2> <p>Current and complete data is <a href="https://www.techtarget.com/searchdatamanagement/feature/Effective-integration-key-to-creating-trusted-data">critical to delivering trustworthy insights</a> for business decision-making. But the extract, transform and load (ETL) integration approach used in traditional data warehouses is often a nonstarter in big data systems, said David Mariani, co-founder and CTO of semantic layer platform vendor AtScale.</p> <p>Batch ETL processes struggle with large data volumes due to data transformation bottlenecks, making it difficult to keep pace with frequent data updates and dynamic analytics requirements. For example, an online retailer's overnight ETL jobs processing millions of daily transactions might exceed their processing windows, leaving business executives and analysts without up-to-date sales data the next morning. ETL is also poorly suited to unstructured or semistructured data because it requires data to be transformed into a predefined schema before loading.</p> <p>The alternative ELT approach addresses these limitations by reversing the load and transform steps. Data is loaded into a data lake or lakehouse in its native format, then transformed and integrated as needed for specific <a href="https://www.techtarget.com/searchbusinessanalytics/feature/8-big-data-use-cases-for-businesses-and-industry-examples">big data use cases</a>. ELT increases scalability, enabling data teams to process large volumes more efficiently and handle high-velocity updates. It also provides greater flexibility for supporting new applications and <a href="https://www.techtarget.com/searchenterpriseai/tip/How-to-identify-and-manage-AI-model-drift">updating AI and analytics models</a> as data changes.</p> <p>Many organizations also deploy real-time data integration and processing technologies to deliver immediate insights in time-sensitive applications such as fraud detection, real-time personalization and operational or patient monitoring. Common real-time technologies include stream processing, event-driven architecture and change data capture. They enable teams to capture data as it's generated or updated and continuously load it into data platforms, often using an ELT approach to structure the data for different applications.</p> </section> <section class="section main-article-chapter" data-menu-title="How AI agents complicate big data integration"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How AI agents complicate big data integration</h2> <p>The <a href="https://www.techtarget.com/searchenterpriseai/tip/How-to-prepare-your-business-for-agentic-AI-adoption">rise of agentic AI</a> further complicates big data integration. Basic integration involves a one-directional pipeline: Data flows from source systems into a repository for analysis. More advanced applications support bidirectional data integration that directly feeds analytics insights back into operational systems. As organizations increasingly <a href="https://www.techtarget.com/searchcio/feature/Agentic-ai-in-practice-lessons-from-real-deployments">deploy AI agents</a>, data teams might need to implement bidirectional integration at a much larger scale.</p> <p>AI agents don't just access and analyze the data in a data lake or lakehouse, said David DuChene, senior manager of data and AI professional services at SHI International. They generate new outputs and surface latent relationships across data domains. Agents also autonomously push enriched insights back to the original source systems if configured to do so. That capability requires more extensive bidirectional integration capabilities, often operating continuously, DuChene said.</p> <p>Data governance pressures also <a href="https://www.techtarget.com/searchdatamanagement/feature/How-agentic-AI-governance-tackles-data-security-challenges">increase with agentic AI</a>. Dan Federoff, vice president and head of data solutions at IT consultancy Bridgenext, said agents don't push back on bad data; they stealthily spread it across an organization. That makes effective data governance -- including strong data quality management and <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-lineage-documentation-imperative-to-data-quality">comprehensive data lineage documentation</a> -- even more critical to successful big data integration in the AI era, Federoff said.</p> <p>In addition, authorization and access control must become more dynamic, said Steve Touw, co-founder and CTO at data security platform vendor Immuta. AI agents operate at machine speed, often across systems and on behalf of multiple users with different permission levels. Touw recommended <a href="https://www.techtarget.com/searchsecurity/tip/Cybersecuritys-agentic-AI-identity-crisis-and-how-to-fix-it">assigning identities to agents</a> that dynamically assume the permissions of different users and configuring the integration layer to provision ephemeral, just-in-time roles. Doing so enables an agent to query data without holding permanent access privileges.</p> <figure class="main-article-image full-col" data-img-fullsize="https://www.techtarget.com/rms/onlineimages/data_management-data_integration.png"> <img data-src="https://www.techtarget.com/rms/onlineimages/data_management-data_integration_mobile.png" class="lazy" data-srcset="https://www.techtarget.com/rms/onlineimages/data_management-data_integration_mobile.png 960w,https://www.techtarget.com/rms/onlineimages/data_management-data_integration.png 1280w" alt="Visual that lists common data integration techniques and methods." height="359" width="560"> <figcaption> <i class="icon pictures" data-icon="z"></i>ELT, change data capture and streaming data integration are commonly used in big data environments. </figcaption> <div class="main-article-image-enlarge"> <i class="icon" data-icon="w"></i> </div> </figure> </section> <section class="section main-article-chapter" data-menu-title="Best practices for big data integration"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Best practices for big data integration</h2> <p>Adopt the following best practices to smooth out the big data integration process and ensure it meets business needs.</p> <h3>Create an all-encompassing data integration strategy</h3> <p>Rick Skriletz, founder and CEO of IT services provider InfoNovus Technologies, said successful integration efforts must work in harmony with several related data management functions:</p> <ul class="default-list"> <li>Data collection across multiple systems.</li> <li>Data processing, storage, security and preparation.</li> <li>Data backup for disaster recovery.</li> </ul> <p>Without a cohesive integration strategy that clearly details how these functions all fit together, data management teams are likely to address each of them separately -- and less effectively, Skriletz said.</p> <h3>Treat data as a product</h3> <p>Data is often viewed as a byproduct of applications and systems. However, a cultural shift toward <a href="https://www.techtarget.com/searchbusinessanalytics/opinion/The-importance-of-data-products">treating data as a product</a> in its own right helps make the integration process more effective, said Jeffrey Pollock, vice president of data replication and streaming products at Oracle. That involves applying product management principles to data assets, with clear ownership, a defined purpose and a focus on data quality, usability and reliability.</p> <p>Netflix has bought into the concept. Tomasz Magdanski, senior manager of data science and engineering at the streaming company, wrote in an October 2025 <a target="_blank" href="https://netflixtechblog.medium.com/data-as-a-product-applying-a-product-mindset-to-data-at-netflix-4a4d1287a31d" rel="noopener">blog post</a> that it's adopting a data-as-a-product framework "to ensure our data assets are managed with the same rigor and strategic focus as traditional products." The goal, he said, is to "elevate data to a first-class entity in our organization's thinking" and ensure that it's trustworthy and aligned with strategic business goals.</p> <h3>Capture critical metadata</h3> <p>Data integration depends on metadata that describes the data elements flowing into a data lake or lakehouse. It's commonly captured in two tools for different users:</p> <ul class="default-list"> <li>A logical data model documents business and technical metadata that data teams use to plan integration work and resolve common problems -- such as data with different structures, inconsistent naming conventions and varying quality levels.</li> <li>A data catalog creates an <a href="https://www.techtarget.com/searchdatamanagement/answer/What-steps-are-key-to-building-a-data-catalog">inventory of data assets with associated metadata</a>, enabling data scientists and business users to find relevant data, understand its context and meaning, and identify datasets that need to be integrated for new use cases.</li> </ul> <h3>Incorporate integration into a data lifecycle management framework</h3> <p>Data lifecycle management (<a href="https://www.techtarget.com/searchstorage/definition/data-life-cycle-management">DLM</a>) establishes policies and procedures for managing data from its creation through archiving and deletion. Data integration should be a core component of an organization's DLM framework. Incorporating it into a systematic framework -- with documented methodologies and governance structures -- is especially critical in big data environments, where volume and complexity make a less rigorous approach unsustainable.</p> <h3>Take an enterprise-wide approach</h3> <p>Maximizing the business value of big data initiatives requires an enterprise-wide approach to data integration rather than isolated implementations in individual departments or business units,&nbsp;according to Faisal Alam, technology consulting leader for the industrial and energy sectors at EY Americas. Siloed integration efforts limit the cross-functional data access and analytics insights needed for strategic decision-making across an organization.</p> <p><b>Editor's note:</b><i> This article was updated in June 2026 for timeliness and to add new information.</i></p> <p><em>George Lawton is a journalist based in London. Over the last 30 years, he has written more than 3,000 stories about computers, communications, knowledge management, business, health and other areas that interest him.</em></p> </section> Data integration in big data systems is even more complex now because of AI. To succeed, it requires a strategy built on new approaches and strong data management. https://cdn.ttgtmedia.com/visuals/searchDataManagement/data_warehouse/datamanagement_article_006.jpg https://www.techtarget.com/searchdatamanagement/feature/Establish-big-data-integration-techniques-and-best-practices Thu, 04 Jun 2026 14:53:00 GMT <p>As AI moves from contained pilots into production and agentic systems, enterprises have learned a hard lesson: the model is rarely the problem. Meaning is, and the missing ingredient is context.</p> <p>Most enterprises have plenty of data. What they lack is a shared, governed layer of meaning that travels with that data wherever it lives. That <a href="https://www.techtarget.com/searchenterpriseai/tip/Exploring-the-context-layer-for-AI-systems">business context layer</a> becomes the foundation for everything else.</p> <section class="section main-article-chapter" data-menu-title="Why context matters"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why context matters</h2> <p>Context is the business meaning wrapped around raw data: the definitions, relationships, rules and lineage that tell a model what a column, a metric, or an entity signifies. An AI system that does not understand what data represents produces fast, fluent and confidently wrong answers at a production scale that <a href="https://www.techtarget.com/searchenterpriseai/feature/How-to-ensure-AI-transparency-explainability-and-trust">erodes trust</a> faster than any accuracy benchmark can rebuild.</p> <p>Context is the difference between a number that signifies customers and a number that signifies quarterly net revenue under the current recognition policy. Without it, "churn" means one thing to finance, another to product and a third to sales. An agent asked about churn will blend those contradictions into outputs no one can act on.</p> </section> <section class="section main-article-chapter" data-menu-title="Context expands scope"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Context expands scope</h2> <p>Context used to sit quietly in a catalog that analysts consulted by hand. Now it must do the following:</p> <ul class="default-list"> <li>Be delivered at runtime to queries, models and autonomous agents acting on data without a <a href="https://www.techtarget.com/searchenterpriseai/feature/Humans-and-AI-The-role-of-people-in-the-new-AI-world">human in the loop</a>.</li> <li>Span fragmented environments across clouds, lakes, SaaS applications and operational systems.</li> <li>Stay governed. An agent that can open tickets, update records and trigger workflows is only as safe as its inherited business rules and guardrails.</li> </ul> <p>An example of a company building for this reality is Starburst. Rather than forcing organizations to centralize data before AI can use it, its Enterprise Intelligence Platform brings AI to the data. Its AI-ready data products combine governed data, metadata and business definitions into reusable, trusted assets that provide consistent business context at runtime regardless of where data resides. Starburst's <a href="https://www.techtarget.com/searchdatamanagement/news/366643641/New-Starburst-platform-extends-AI-to-distributed-data">AI Data Assistant</a> then delivers that context where business users work -- inside applications, workflows and agents -- so meaning follows the question instead of rebuilding it for every model.</p> <p>But this strategic point is bigger than any one vendor. Context, not compute, is becoming the gating factor for production AI. Organizations continuing to layer models on fragmented, undefined data will keep receiving fast answers they cannot trust, while those investing in governed, distributed context will scale AI with confidence.</p> <p><em>Stephen Catanzano is a senior analyst at Omdia, where he covers data management and analytics.</em></p> <p><em>Omdia is a division of&nbsp;Informa TechTarget.&nbsp;Its analysts have business relationships with technology vendors.</em></p> </section> Your AI model might not be the reason it fails in production. Missing or unclear context throws off the model, leading to untrustworthy outputs that the business can't act on. https://cdn.ttgtmedia.com/visuals/digdeeper/2.jpg https://www.techtarget.com/searchdatamanagement/opinion/Context-is-the-make-or-break-layer-for-AI-in-production Thu, 04 Jun 2026 10:58:00 GMT <p>Most data problems do not announce themselves.</p> <p>A pipeline runs, rows of data arrive, dashboards refresh -- but somewhere along the line, a feature quietly starts behaving differently than it did last week. By the time anyone notices, an AI model has been making faulty predictions for days.</p> <p>Data observability catches these silent failures by applying site reliability engineering principles to data: instrumentation, monitoring and alerting.</p> <section class="section main-article-chapter" data-menu-title="What is data observability?"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What is data observability?</h2> <p>Data observability is the continuous, automated monitoring of data's health <a href="https://www.techtarget.com/searchdatamanagement/tip/Top-data-observability-use-cases">as it moves through pipelines</a> and into models. Rather than waiting for downstream users to spot problems, observability tools watch the data and compare today's arrivals with recent history -- or with what the schema and data contract specify should arrive. When something is off, they raise an alert close to the source, ideally before bad data reaches a model or a user.</p> <p>The discipline is often described in five pillars:</p> <ul class="default-list"> <li><b>Freshness.</b> Is the data up to date?</li> <li><b>Volume. </b>Are row counts within the expected range?</li> <li><b>Schema</b>. Has the data's structure changed?</li> <li><b>Distribution. </b>Do the data values fall within expected boundaries?</li> <li><b>Lineage.</b> What upstream and downstream assets are affected by data issues?</li> </ul> <p>Together, they answer the basic question: Is this data still trustworthy?</p> </section> <section class="section main-article-chapter" data-menu-title="Why AI projects need observability more than traditional analytics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why AI projects need observability more than traditional analytics</h2> <p>A broken dashboard is annoying. A broken feature pipeline is dangerous. Machine learning (ML) models do not throw errors when their input data shifts. They keep producing predictions, <a href="https://www.techtarget.com/searchdatamanagement/tip/Experts-share-practices-to-overcome-AI-data-readiness">just inaccurate ones</a>. A categorical feature that suddenly has a new value, a numeric column whose mean has crept upward, a timestamp column with growing nulls: Any of these can silently degrade a model for weeks.</p> <p>Observability is foundational for detecting concept drift and <a href="https://www.techtarget.com/searchitoperations/feature/Meeting-the-challenges-of-scaling-AI-with-MLOps">training-serving skew</a>. Comparing live feature distributions against the training set -- automatically and continuously -- provides reliable signals that an ML model still operates in the world it was trained for. Without observability, teams must rely on lagging business metrics that reveal problems only after they've caused damage.</p> </section> <section class="section main-article-chapter" data-menu-title="What good data observability captures"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What good data observability captures</h2> <p>A mature data observability practice does more than uptime checks. Effective systems also:</p> <ul type="disc" class="default-list"> <li>Trigger alerts about stalled upstream jobs or other data freshness issues in minutes, not days.</li> <li>Track expected volume ranges, accounting for daily and seasonal patterns.</li> <li>Monitor for added columns, type changes, new enum values and other schema changes.</li> <li>Compare live data against historical baselines to check value distributions.</li> <li>Measure<b> </b>null and uniqueness rates &nbsp;to catch shifts that often precede failures.</li> <li>Use statistical tests tuned for each model's inputs to identify feature-level drift.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="How to start with data observability"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How to start with data observability</h2> <p>Begin with the data assets your models depend on most directly: training tables, feature views and inference inputs.</p> <p>Add basic freshness and volume checks first, since they catch the most incidents with the least effort.</p> <p>Layer in schema and distribution monitoring as you learn what normal looks like.</p> <p>Use tools to <a href="https://www.techtarget.com/searchdatamanagement/feature/4-trends-that-will-shape-data-management-and-AI-in-2026">streamline the process</a> -- whether it's specialized technologies such as Monte Carlo, Soda and Great Expectations or the observability features built into modern data platforms and catalogs. The principle is the same regardless of tooling: Monitor near the data source, alert the producing team when problems are detected and route incidents to a data owner who can fix the issue.</p> <p>But avoid alert fatigue from the start. Tune thresholds against real history rather than guesses. Treat noisy checks like noisy production alerts: Fix them or remove them.</p> </section> <section class="section main-article-chapter" data-menu-title="How observability pays off by catching drift"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How observability pays off by catching drift</h2> <p>Data observability turns silent failures into loud ones and shifts incident response from forensic to preventive. Combined with data contracts and lineage, it <a href="https://www.techtarget.com/searchdatamanagement/opinion/Tracing-data-lineage-in-AI-systems">closes the loop</a>: Contracts define what data should be, lineage shows how it flows, and observability confirms that reality matches the agreement. For AI projects, where models can degrade invisibly and expensively, that <a target="_blank" href="https://www.gartner.com/en/newsroom/press-releases/2026-05-12-gartner-predicts-40-percent-of-organizations-deploying-ai-will-use-ai-observability-to-monitor-model-performance-by-2028" rel="noopener">verification</a> is key to making production trustworthy.</p> <p><i>Stephen Catanzano is a senior analyst at Omdia, where he covers data management and analytics.</i></p> <p><i>Omdia is a division of Informa TechTarget. Its analysts have business relationships with technology vendors.</i></p> </section> Data quality issues get amplified in AI applications. Models can produce confident -- but misleading -- forecasts and conclusions without observability safeguards. https://cdn.ttgtmedia.com/visuals/digdeeper/3.jpg https://www.techtarget.com/searchdatamanagement/opinion/Data-observability-for-AI-helps-curb-poor-model-performance Tue, 26 May 2026 13:47:00 GMT <p>A well-executed big data strategy helps enterprises improve operational performance, optimize marketing campaigns and prioritize product development plans. But data leaders face various challenges in advancing <a href="https://www.techtarget.com/searchdatamanagement/The-ultimate-guide-to-big-data-for-businesses">big data initiatives</a> from boardroom discussions to successful deployments.</p> <p>Data teams must work with IT to build an infrastructure that collects diverse data from numerous sources and makes it available for use in analytics and AI applications. They also need to ensure big data systems meet performance, scalability and timeliness requirements, with high data quality and strong <a href="https://www.techtarget.com/searchdatamanagement/definition/data-governance">data governance</a> controls -- while also controlling implementation costs.</p> <p>Perhaps most importantly, data leaders must engage with business executives to determine&nbsp;<a href="https://www.techtarget.com/searchbusinessanalytics/feature/6-big-data-benefits-for-businesses">how big data can benefit the organization</a>&nbsp;and align the strategy with key business goals and priorities.</p> <p>Looking more deeply at these issues, here are 10 common big data challenges, along with advice on overcoming them.</p> <section class="section main-article-chapter" data-menu-title="1. Managing large volumes of data"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Managing large volumes of data</h2> <p>Big data typically involves large volumes of data from disparate systems, applications and external sources. It also usually includes a mix of structured, unstructured and semistructured data, which is often created or updated at a fast pace. Managing this combination of volume, variety and velocity -- the traditional 3 V's of big data -- is inherently complicated.</p> <p>That starts with extracting and consolidating relevant data from all the different sources -- CRM and ERP systems, website and application logs, sensors, social networks and more -- into a <a href="https://www.techtarget.com/searchdatamanagement/feature/Building-a-big-data-architecture-Core-components-best-practices">unified big data architecture</a>. Such architectures commonly have been built on data lakes, scalable platforms that store diverse types of data. But Donald Farmer, principal at consulting firm TreeHive Strategy, said many data lakes are more like swamps, with sprawling data sets that are difficult to track and manage effectively.</p> <p>Farmer added that <a href="https://www.techtarget.com/searchdatamanagement/news/366545117/Lakehouse-architecture-the-best-fit-for-modern-data-needs">newer data lakehouse platforms</a> help ease those issues by combining the scalability and storage flexibility of data lakes with the more rigorous data management functions of traditional data warehouses. For example, he said Apache Iceberg and other open table formats provide transactional consistency and data versioning in data lakehouses, enabling data management teams to maintain audit trails and modify schemas without disrupting analytics and AI applications.</p> </section> <section class="section main-article-chapter" data-menu-title="2. Finding and fixing data quality issues"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Finding and fixing data quality issues</h2> <p><a href="https://www.techtarget.com/searchenterpriseai/Ultimate-guide-to-artificial-intelligence-in-the-enterprise"></a>Big data applications produce bad results when data quality issues affect systems. These issues become more significant -- and harder to address -- as data management and analytics teams ingest more and more data. Monitoring data quality, identifying problems and fixing them is a continuous process, Bunddler CEO Paul Kovalenko said.</p> <p>Bunddler, an online marketplace for finding shopping assistants who help people buy products and arrange international shipments, experienced that firsthand as it scaled to 500,000 customers. The New York-based company uses big data to provide a highly personalized UX, monitor trends and identify upselling opportunities for assistants, but <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-quality-for-big-data-Why-its-a-must-and-how-to-improve-it">effective data quality management</a> is a pressing concern.</p> <p>Duplicate entries and typos are common in the data Bunddler collects from various sources, Kovalenko said. To root out such problems, it created a tool that matches duplicates with minor data differences and flags potential typos. Higher-quality data from using the tool has increased the accuracy of analytics insights, he said.</p> <p>AI can also help organizations improve data quality: It's increasingly being used to validate data and detect anomalies, errors, inconsistencies and other quality issues.</p> </section> <section class="section main-article-chapter" data-menu-title="3. Dealing with data integration complexities"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Dealing with data integration complexities</h2> <p>While big data platforms enable organizations to collect and store large amounts of varied data, the&nbsp;<a href="https://www.techtarget.com/searchdatamanagement/feature/Big-data-collection-processes-challenges-and-best-practices">data collection process</a>&nbsp;is challenging, said Rosaria Silipo, a data scientist, author and co-host of the "My Data Guest" podcast. In particular, integrating sets of big data is more complex than conventional data integration due to the different types of data involved and the fast pace of updates.</p> <p>Data leaders and teams need to think through their organization's data integration requirements upfront. Ad hoc integration for specific projects often results in redundant efforts and substantial rework of integration scripts or routines, Silipo said. Optimizing the ROI of big data investments requires a <a href="https://www.techtarget.com/searchdatamanagement/feature/Establish-big-data-integration-techniques-and-best-practices">strategic approach to data integration</a>, she added.</p> <p>That typically involves extract, load and transform (ELT) processes rather than the traditional ETL ones used in data warehouses. ELT loads data into a data lake or lakehouse in its native format, then combines and transforms it as needed for specific use cases. Real-time integration is also common in big data environments, and the growing adoption of AI tools and agents is accelerating a shift from rigid data pipelines to flexible architectures that <a href="https://www.techtarget.com/searchbusinessanalytics/news/365534255/Data-pipelines-deliver-the-fuel-for-data-science-analytics">deliver data to applications more dynamically</a>.</p> </section> <section class="section main-article-chapter" data-menu-title="4. Scaling big data systems efficiently and cost-effectively"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Scaling big data systems efficiently and cost-effectively</h2> <p>Enterprises waste a lot of money collecting and storing big data if they don't have scalable systems capable of handling both current and future processing workloads. As a result, data teams should map out planned uses and required data types and schemas before designing and deploying big data systems.</p> <p>But that's easier said than done, said Travis Rehl, CTO and head of product at data, AI and cloud services provider Innovative Solutions. "Oftentimes, you start from one data model and expand out, but quickly realize the model doesn't fit your new data points -- and you suddenly have technical debt you need to resolve," Rehl said.</p> <p>Appropriate data structures make it easier to reuse data efficiently. For example, Parquet files often provide a better performance-to-cost ratio than CSV dumps within a data lake or lakehouse. Consistent <a href="https://www.techtarget.com/searchstorage/tip/Data-retention-and-destruction-policy-template-A-free-download">retention policies</a> cycle out old data from repositories as its analytics value erodes. When latency is an issue, teams also need to consider whether to run systems in the cloud, in on-premises data centers or <a href="https://www.techtarget.com/searchdatacenter/tip/From-core-to-edge-Strategies-for-scalable-compliant-and-agile-IT">on edge servers</a>, while balancing performance with deployment and management costs.</p> </section> <section class="section main-article-chapter" data-menu-title="5. Evaluating and selecting big data technologies"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. Evaluating and selecting big data technologies</h2> <p>Data leaders and their teams can choose from a <a href="https://www.techtarget.com/searchdatamanagement/feature/15-big-data-tools-and-technologies-to-know-about">wide range of big data technologies</a> that often overlap in capabilities. Both open source tools and commercial platforms are available, further complicating the evaluation and selection process. Making the right choices is critical to gaining the expected business benefits from big data initiatives.</p> <p>To help inform technology decisions, teams should consider current and future data needs for both batch processing and real-time streaming from different sources. The <a href="https://www.techtarget.com/searchbusinessanalytics/definition/data-preparation">data preparation</a> capabilities required to support AI, machine learning and other advanced analytics applications should also be assessed, as well as where data will be processed and stored. The ability to easily update analytics and AI models in data platforms is another key consideration.</p> </section> <section class="section main-article-chapter" data-menu-title="6. Generating valuable business insights"> <h2 class="section-title"><i class="icon" data-icon="1"></i>6. Generating valuable business insights</h2> <p>The volume and complexity of big data complicate efforts to analyze and use it. Organizations often struggle to generate valuable insights and apply them in business operations in an impactful way, said Bill Szybillo, manager of BI engineering at firearms maker Sig Sauer Inc.</p> <p>Doing so requires a clear understanding of the data's business context and <a href="https://www.techtarget.com/searchbusinessanalytics/feature/8-big-data-use-cases-for-businesses-and-industry-examples">potential use cases</a>. But Silipo said she has found that many data leaders and teams focus on the technology and pay less attention to how big data systems can be used to achieve desired business outcomes.</p> <p>Teams that don't work with the people closest to business problems when planning data platforms, pipelines and storage architectures might build technically sound systems that produce little business value. Pilot projects are useful not only for engaging business users from the start, but also for surfacing limitations early on in big data initiatives and delivering some quick wins to demonstrate business benefits.</p> </section> <section class="section main-article-chapter" data-menu-title="7. Hiring and retaining workers with big data skills"> <h2 class="section-title"><i class="icon" data-icon="1"></i>7. Hiring and retaining workers with big data skills</h2> <p>Finding workers with the required skills is another common challenge -- and growing AI use adds new requirements for expertise in designing, training and supervising AI models. But data scientists and other analytics professionals with AI skills are in high demand, as are data engineers and workers skilled in <a href="https://www.techtarget.com/searchdatamanagement/feature/Building-a-strong-data-analytics-platform-architecture">deploying and managing data platforms</a>.</p> <p>In addition, technical skills alone aren't enough. Data teams also must be able to identify risks, manage internal expectations and resolve issues, said Pablo Listingart, founder and executive director of ComIT and Comunidad IT, charitable organizations that provide free IT training programs in Canada and Argentina. "Many big data initiatives fail because of incorrect expectations and faulty estimations that are carried forward from the beginning of the project to the end," he noted.</p> <p>Vojtech Kurka, co-founder and head of R&amp;D at customer data platform vendor Meiro, said creating the right culture helps attract and retain skilled workers. Kurka initially thought Meiro could solve its data problems with simple SQL and Python scripts. But he later realized that to meet its goals, the company needed to hire people with more advanced data skills and keep them satisfied and motivated.</p> <p>Organizations can also partner with providers of AI, analytics, data management and software development services to fill big data skills gaps. In some cases, that's faster and less expensive than hiring new employees. But data leaders should carefully evaluate a provider's costs and capabilities and assess whether internal hiring is a better long-term option.</p> </section> <section class="section main-article-chapter" data-menu-title="8. Keeping costs from getting out of control"> <h2 class="section-title"><i class="icon" data-icon="1"></i>8. Keeping costs from getting out of control</h2> <p>Another common challenge is avoiding what David Mariani, co-founder and CTO of semantic layer platform vendor AtScale, called the "cloud bill heart attack."</p> <p>Many enterprises use existing data consumption metrics to estimate the computing costs of new big data infrastructure, but expanded access to richer, more granular data sets often increases user demand for computing resources. Cloud systems that elastically scale to handle higher data processing and analysis workloads will drive up costs unexpectedly if companies underestimate their resource needs.</p> <p>On-demand pricing models can also increase costs if the use of big data systems isn't managed effectively.&nbsp;Fixed-resource pricing alleviates that problem, but doesn't completely solve it: Poorly written applications that consume excessive resources block other workloads from running if the specified usage limit is reached. "I've seen several customers where users have written $10,000 queries due to poorly designed SQL," Mariani said. Data teams need to implement fine-grained query controls to prevent that.</p> <p>Rehl said data leaders should also raise the cost issue upfront with business and data engineering teams when planning big data deployments to ensure organizations budget appropriately for required computing resources and include effective cost controls.</p> </section> <section class="section main-article-chapter" data-menu-title="9. Governing big data environments"> <h2 class="section-title"><i class="icon" data-icon="1"></i>9. Governing big data environments</h2> <p>Without effective data governance, "much of the benefit of broader, deeper data access can be lost," Mariani said. But data governance issues become harder to address as big data applications expand across systems. Cloud architectures that make it more feasible for enterprises to collect and store ever-increasing volumes of raw, unaggregated data compound governance challenges.</p> <p>Lax data governance reduces the accuracy of analytics and AI outputs and allows protected information to creep into applications that shouldn't include it, creating compliance risks. In addition to the <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know">data protection and privacy laws</a> that mandate strong governance, <a href="https://www.techtarget.com/searchenterpriseai/feature/AI-regulation-What-businesses-need-to-know">AI regulations</a> are becoming a factor, Farmer said. For example, under the EU AI Act, qualifying organizations deploying AI systems classified as high-risk must meet a set of data governance and management requirements starting in August 2026.</p> <p>Investing time upfront to identify and&nbsp;manage big data governance issues&nbsp;makes it easier to provide self-service data access without requiring direct oversight of each new use case. Treating data as a product with built-in governance rules also helps prevent usage and compliance issues.</p> </section> <section class="section main-article-chapter" data-menu-title="10. Ensuring that AI tools produce trustworthy results"> <h2 class="section-title"><i class="icon" data-icon="1"></i>10. Ensuring that AI tools produce trustworthy results</h2> <p>Generative AI (GenAI) and agentic AI tools <a href="https://www.techtarget.com/searchdatamanagement/feature/How-agentic-AI-governance-tackles-data-security-challenges">amplify data management and governance issues</a> in big data systems. For example, AI agents configured to autonomously monitor, analyze and act on data can create cascading errors and compliance problems without proper oversight.</p> <p>Comprehensive training and ongoing supervision are required to ensure that AI's actions are accurate, unbiased and trustworthy, said Michael O'Malley, senior vice president of strategy and growth at Customer Analytics LLC, an AI, analytics and data engineering services provider. "Agents and generative AI are powerful tools," O'Malley said. "But just owning an expensive hammer doesn't make you a master carpenter."</p> <p>Data quality is also a key consideration: An AI agent is only as reliable as the data it analyzes, Silipo noted. In addition, the models that underpin GenAI and agentic AI tools must be updated when new business trends or scenarios inevitably emerge. Otherwise, the tools won't be able to adapt, leading to flawed analytics and actions.</p> <p><strong>Editor's note:</strong> <em>This article was updated in May 2026 for timeliness and to add new information.</em></p> <p><em>George Lawton is a journalist based in London. Over the last 30 years, he has written more than 3,000 stories about computers, communications, knowledge management, business, health and other areas that interest him.</em></p> </section> Big data initiatives pose various challenges for data leaders and teams. Here are 10 common ones and advice on overcoming them to ensure deployments are successful. https://cdn.ttgtmedia.com/visuals/searchCloudProvider/service_management/cloudprovider_article_008.jpg https://www.techtarget.com/searchdatamanagement/tip/10-big-data-challenges-and-how-to-address-them Fri, 22 May 2026 22:40:00 GMT <p>As self-service analytics blooms and teams prioritize speed over consistency, dashboard sprawl can take root, sowing confusion among users and adding extra costs and work.</p> <p>What begins as convenience can quietly become complicated: Duplicate tools drive licensing and platform overhead, competing data definitions erode trust, and leaders struggle to show the benefits of their analytics investments.</p> <p>As organizations struggle with dashboard sprawl challenges, data teams are increasing spending, which can compound the problem. Mordor Intelligence predicts the global data analytics market will reach $108.79 billion in 2026, up from $82.33 billion in 2025. In many organizations, dashboards multiply faster than governance rules, ownership assignments and internal standards can <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">keep up to maintain consistent metrics </a>across the enterprise.</p> <section class="section main-article-chapter" data-menu-title="When dueling dashboards create doubt"> <h2 class="section-title"><i class="icon" data-icon="1"></i>When dueling dashboards create doubt</h2> <p>Matt Arellano has seen what happens when dashboards outpace governance.</p> <p>Arellano, a senior vice president at IT services and consulting company Genpact, worked with a global retailer whose U.S. and European sales teams used separate analytics systems with dashboards that calculated gross sales differently. As a result, the teams <a href="https://www.techtarget.com/searchdatamanagement/feature/Controlling-data-sprawl-requires-governance-discipline">reported inconsistent numbers</a> to the finance department, "leaving a numerical problem for finance to figure out," Arellano said.</p> <p>The conflicting calculations also meant the company didn't spend its marketing dollars efficiently and made it appear that the U.S. and European markets were performing differently, even though they were not, Arellano said.</p> <p>Organizations have invested heavily in data platforms and business intelligence (BI) technologies , betting that analytics would yield clearer insights and, ultimately, <a href="https://www.techtarget.com/searchbusinessanalytics/feature/Real-time-edge-analytics-use-cases-for-business">faster, more informed business decisions</a>. Yet trust in analytics results is slipping in many organizations, as is the value of data analysis, often due to dashboard sprawl.</p> <p>The problem is widespread, said Boris Evelson, a vice president and principal analyst at Forrester Research.</p> <p>"The majority of organizations are still in the broken dashboard phase," Evelson said. "And the elephant in the room here is that as a result, no one can figure out what their ROI is on their data and analytics investments. This is one of the top complaints we hear [from clients]. They don't know how much they've gained for all their work. They're saying, 'We've got all these dashboards, but how do we know they're bringing us business value?'"</p> </section> <section class="section main-article-chapter" data-menu-title="Why dashboard sprawl is rising"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why dashboard sprawl is rising</h2> <p>The typical organization has numerous dashboards. Several factors have contributed to their growth, according to experts.</p> <p>First is the <a href="https://www.techtarget.com/searchbusinessanalytics/feature/The-future-of-business-intelligence-Top-trends">democratization of data</a>, which extended access to data to employees across organizations rather than limiting it to skilled analytics professionals, such as BI developers and data scientists.</p> <p>Second is the large enterprise investment in analytics tools that include dashboards.</p> <p>Third is the growing availability of technologies that enable self-service users to easily create their own dashboards. As a result, many dashboards are built outside formal governance channels, which can lead to shadow analytics. Some dashboards are built by a particular user or team, used once and forgotten, becoming so-called zombie dashboards that burn up computing resources and licensing budgets without any ROI.</p> <p>"The democratization of data and business intelligence, along with the low barrier of entry for creating new dashboards, has accelerated the sheer number of dashboards and reports that are produced and consumed," said Nick Kramer, principal of applied solutions at consultancy SSA &amp; Company.</p> <p>But it's not just the number of dashboards within an organization that's a problem, Kramer and others said. The bigger issue is inconsistent data definitions across dashboards -- a problem that stems from <a href="https://www.techtarget.com/searchdatamanagement/feature/AI-data-governance-guidance-that-gets-you-to-the-finish-line">governance gaps</a>.</p> <p>"Companies may have data governance, but they don't extend it into their BI layer," Kramer said. "And you have a ton of dashboards being built, and they're also relatively ungoverned. So, you have different dashboards with an overlap in what they report, and they do it differently."</p> </section> <section class="section main-article-chapter" data-menu-title="How dashboard sprawl hurts organizations"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How dashboard sprawl hurts organizations</h2> <p>Dashboard sprawl negatively impacts an organization in multiple ways.</p> <p>To start, dashboard sprawl adds unnecessary costs, said Doug Leal, vice president of data and analytics consulting at CGI, an IT and business consultancy. Duplicate tools and zombie dashboards <a href="https://www.techtarget.com/searchcio/feature/Predictable-IT-spending-in-an-unpredictable-economy">incur excess licensing fees</a>, require IT support and consume compute resources without providing sufficient -- or sometimes any -- benefits to justify the money spent on them.</p> <p>Dashboard sprawl also has indirect costs, Leal explained. Because data definitions vary across multiple dashboards, different workers and teams produce conflicting reports. That leads to disputes over who has the right insights.</p> <p>"There's an inability to make a decision. That leads to a lack of agility, and you become stagnant as an organization because you can't move forward," Leal said.</p> <p>"The paralysis that comes with not knowing what the real answer is, is real," Kramer added.</p> <p>That, in turn, wears away employee and executive trust in the organization's data, even if the <a href="https://www.techtarget.com/searchdatamanagement/opinion/Most-data-governance-wasnt-built-for-AI">data quality</a> itself is sound.</p> <p>"When you have multiple dashboards reporting the same metric, but they're not reporting the same number, [people] then don't trust the data. And the cost to turn that around and get them to trust the data again is huge," Leal said.</p> </section> <section class="section main-article-chapter" data-menu-title="How leaders can rein in dashboard sprawl"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How leaders can rein in dashboard sprawl</h2> <p>Dashboard sprawl is not an indictment against dashboards themselves, nor even having a high number of them in an organization.</p> <p>"Dashboards absolutely bring value. They are absolutely useful, and we cannot operate without them," Evelson said. "But there are too many dashboards created with expensive dollars that aren't being used or that are being used that provide signals without insight, so you can't act on them. That's the problem."</p> <p>Evelson stressed that an organization shouldn't eliminate dashboards just to reduce the number; instead, it should aim to better govern them.</p> <p>The goal of better governance is to reduce duplicate tools, <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Top-11-business-intelligence-challenges-and-how-to-overcome-them">enforce consistent data definitions</a> and data quality, and retire dashboards when they're no longer needed or no longer provide value.</p> <p>Those are challenging tasks, experts said.</p> <p>To execute them effectively, experts said organizations must have a strong data governance operating model that includes metadata management and a semantic <a target="_blank" href="https://www.youtube.com/watch?v=K_VPF9URZo4" rel="noopener">layer</a>, and that enforces data stewardship and ownership. They also need asset management to inventory dashboards and a searchable catalog so teams can reuse existing ones.</p> <p><em>Mary K. Pratt is an award-winning freelance journalist specializing in enterprise IT, cybersecurity strategy and data management.</em></p> </section> The proliferation of dashboards, coupled with conflicting data definitions, exposes governance issues in organizations and reduces the ROI of analytics investments. https://cdn.ttgtmedia.com/rms/onlineimages/collab_a120705009.jpg https://www.techtarget.com/searchbusinessanalytics/feature/How-dashboard-sprawl-challenges-upend-enterprise-analytics Fri, 22 May 2026 13:32:00 GMT <p>Data sovereignty has expanded from a regulatory compliance matter into a broader risk management challenge as IT environments and global pressures grow more complex.</p> <p>Addressing data sovereignty once meant complying with the laws governing data from its country of origin to others where it's processed or stored. But sovereignty concerns now also encompass AI models, supply chain disruptions and military conflicts. Chief data officers, CIOs and CISOs must consider the expanding scope of sovereignty and <a href="https://www.techtarget.com/searchenterpriseai/tip/How-to-navigate-data-sovereignty-for-AI-compliance">how to navigate a multitude of variables</a> subject to rapid, unpredictable change. Sovereignty strategies need to maintain control and flexibility across jurisdictions while also balancing cost, compliance and performance.</p> <p>Experts recommend aligning sovereignty goals with data sensitivity levels and treating sovereignty as a risk management endeavor. The results, ideally, will not only meet regulatory requirements but also strengthen organizational resilience and technology independence.</p> <section class="section main-article-chapter" data-menu-title="The new world of data sovereignty"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The new world of data sovereignty</h2> <p>The task for enterprise data leaders is challenging. On the traditional compliance side, they face numerous regulations that address elements of sovereignty but do not fully define it. Organizations navigating jurisdictional control of data do so without a single, authoritative global standard comparable to the EU's GDPR, which serves as a de facto data privacy standard.</p> <p>"A clear GDPR for sovereignty doesn't exist," said Dario Maisto, a senior analyst at Forrester. "There is no legislation whatsoever in the world that tells you what sovereignty is and isn't."</p> <p>Maisto said that some regulations, such as India's Digital Personal Data Protection Act, 2023, touch on sovereignty within specific domains but don't define it more broadly. The <a href="https://www.computerweekly.com/news/366549932/India-gets-ready-for-new-data-protection-regime">DPDPA</a> applies only to digital personal data or data digitized after being collected, he noted. It focuses on data protection and permits cross-border data flows, except to restricted regions, Maisto added.</p> <p>Another complication for data sovereignty efforts is physical risk to data centers and other technology infrastructure. Geopolitical conflict and natural disasters can strip organizations of jurisdictional control even when data is stored and processed in accordance with local laws.</p> <p>For example, Iran on Feb. 28 began launching salvos of drone and missile attacks at the Gulf States in response to the U.S. and Israeli strikes against the country. The next day, AWS reported damage to data centers in Bahrain and the United Arab Emirates caused by Iranian <a target="_blank" href="https://health.aws.amazon.com/health/status" rel="noopener">drones</a>. The attacks severely disrupted services at the facilities, and the cloud provider urged customers to migrate workloads to other AWS Regions in the U.S., Europe or Asia-Pacific, "as appropriate for your latency and data residency requirements."</p> <p>"The geopolitics have shifted tremendously in the last few months," said Ron Babin, adjunct research advisor for IDC's IT Executive Programs. "As a CIO, that's something that you rarely have had to worry about -- having your data center as a target of military forces."</p> <p>The threat to IT infrastructure and data, <a href="https://www.computerweekly.com/feature/Breaking-the-stranglehold-Responses-to-data-sovereignty-risk">concerns about dependencies</a> on foreign tech providers, and ambiguous regulations have drastically changed the nature of sovereignty.</p> <p>"There is a paradigm shift going on in the market," Maisto said. "It used to be a data protection problem [and] a privacy-related issue. Now, instead, it is really about risk management."</p> </section> <section class="section main-article-chapter" data-menu-title="Developing a sovereignty strategy"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Developing a sovereignty strategy</h2> <p>Against this backdrop, industry executives and analysts outlined frameworks and methods to help IT leaders operationalize sovereignty strategies.</p> <p>Kyndryl, an IT infrastructure firm, in April launched a suite of advisory, implementation and managed services focused on sovereignty. Logan Wolfe, partner for global enterprise transformation, AI and sovereign tech strategy at Kyndryl, said the services reflect the new directions of sovereignty.</p> <p>"The conversation has shifted, fundamentally so, from a regulatory compliance basis -- a box-ticking exercise, if you will -- to how can we apply [sovereignty principles and strategy] as a fundamental risk framework to operate resiliently," he said.</p> <p>The Kyndryl framework spans three dimensions: data sovereignty, operational sovereignty and technological sovereignty. The data side focuses on where data is stored and processed and <a href="https://www.techtarget.com/searchapparchitecture/tip/Privacy-compliance-and-governance-are-changing-development">who can access it</a>, Wolfe said. It also covers which regulatory regimes apply to a particular organization. Operational sovereignty involves an organization's ability to operate, maintain and recover systems without undue external dependencies. Technological sovereignty centers on limiting its dependence on technologies controlled or subject to interference by foreign governments.</p> <p>A readiness assessment evaluates a business's posture across the three dimensions. This should spur a strategic discussion and produce an action plan on how to achieve specific sovereignty goals. Wolfe said those goals have different drivers, such as data and AI, operational independence and regulatory compliance. Each goal will have a different action plan.</p> <p>IDC's framework focuses on sovereign AI, which includes data sovereignty. The framework <a target="_blank" href="https://www.idc.com/resource-center/blog/ai-sovereignty-risk-a-five-step-agenda-for-cios/" rel="noopener">covers</a> core technologies such as data models, infrastructure and AI chips and extends to include sourcing of model training data, supply chain relationships and regulatory compliance. Geopolitics is included as a constraint.</p> <p>Babin said CIOs, CDOs and other leaders need to determine where their organization stands regarding the jurisdictions where it operates and the applicable rules and regulations. That means understanding "the footprint of your organization, [including] customers, operating units and supply chains," he noted.</p> <p>In addition, the organization's industry -- whether financial services, transportation, power generation or others -- will have specific AI regulations to consider, Babin said.</p> <p>At Forrester, Maisto also cited <a href="https://www.techtarget.com/whatis/feature/Sovereign-AI-explained">AI's influence on data sovereignty approaches</a>. Data resides in an infrastructure, flows through a network, is used by SaaS offerings and feeds into AI workloads and large language models, he said. At that point, technology and business users operationalize the data. <br><br>"That's where data sovereignty actually develops into six different domains: data, infrastructure, network, software, AI and people," Maisto said.</p> <p>Across those domains, the specific sovereignty issue will drive a particular remedy, he added.</p> </section> <section class="section main-article-chapter" data-menu-title="Implementing sovereignty controls"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Implementing sovereignty controls</h2> <p>The strategy and planning work leads to the implementation stage, which focuses on controls -- the risk mitigation measures used to address sovereignty challenges.</p> <p>Cloud platform infrastructure, which houses data and workloads, is a critical area for establishing controls. The sovereign instinct might be to keep all data assets within a locally owned and hosted cloud.</p> <p>"A lot of people look at sovereignty and they think isolationism: 'We are just going to isolate as much as possible and create this digital fortress,'" Wolfe said.</p> <p>But instead of dropping public clouds entirely, an organization can pursue segmentation as a design pattern. A business in the EU, for instance, could identify and <a href="https://www.techtarget.com/searchsecurity/tip/Confidential-computing-use-cases-that-secure-data-in-use">segment sensitive data sets</a> and workloads restricted to EU jurisdiction with tighter operational controls, Wolfe said. Other data sets and workloads could continue to benefit from global cloud scale, he added.</p> <p>For example, a business might want to adopt or continue using a SaaS product that only runs on a particular hyperscaler's platform. Such cases make achieving full sovereignty a nightmare, Maisto said. Businesses operating in more than 100 countries, as some of his clients do, underscore the impossibility of the task, he added.</p> <p>Maisto's alternative is what he terms "minimum viable sovereignty," which relies on workload assessment. Here, data and IT leaders identify workloads that absolutely require <a href="https://www.forrester.com/blogs/digital-sovereignty-why-tech-execs-must-act-now/">sovereign controls</a>. The idea is to reserve the tightest control for the most sensitive workloads<i>, </i>while avoiding over-engineered protection for less sensitive ones.<b><i> </i></b>This method offers the ability to combine hyperscaler clouds, local cloud providers, private clouds and on-premises resources as appropriate.</p> <p>With minimum viable sovereignty, organizations must determine whether a sovereign cloud is an available option for their problems, Maisto noted. Sovereign capabilities from hyperscalers and other cloud providers are <a href="https://www.computerweekly.com/feature/Sovereign-cloud-and-AI-services-tipped-for-take-off-in-2026">entering the market</a> at a quickening pace, so leaders need to check regularly for new developments.</p> <p>If a sovereign cloud is available, the assessment shifts to cost. Maisto outlined some questions to consider: Is the sovereign offering more or less expensive than the non-sovereign version? If it's more expensive, does the sovereign option justify the cost?</p> <p>In addition, organizations should weigh cost trade-offs between hyperscalers and local providers. Achieving sovereignty with a hyperscaler might prove more costly than working with a local vendor, Maisto said. But even if the local vendor is cheaper, change management and migration costs would still need to be considered, he noted.</p> <p>Controls such as <a href="https://www.techtarget.com/searchdatabackup/tip/The-new-standards-for-cyber-resilient-backup-strategy">data vaulting and advanced encryption</a> also play a role in building a sovereign architecture. With data vaulting, isolated and protected copies are created for resilient recovery, Wolfe said. Advanced encryption coupled with customer-controlled key management provides sovereign control over key access, he added.</p> </section> <section class="section main-article-chapter" data-menu-title="Building AI models with sovereignty in mind"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Building AI models with sovereignty in mind</h2> <p>As AI becomes embedded in the bigger sovereignty picture, enterprises need to build AI models without moving raw data around by using techniques such as federated learning and secure multiparty computation. Fariba Wells, senior vice president of global government affairs and policy at Kyndryl, said she has seen some uptake for those methods.</p> <p>"Adoption is growing but remains selective -- concentrated in highly regulated sectors where data movement is tightly constrained," she said.</p> <p>Wells said the techniques align well with sovereignty objectives in that they allow AI models to be trained or refined without centralizing raw data -- directly reducing legal and regulatory exposure.</p> <p>"Federated learning is the more mature of the two in enterprise settings," Wells noted.</p> <p>That approach enables model training within local or sovereign environments while sharing only model updates or parameters across regions, she said. This lets organizations benefit from broader data patterns without violating data residency or access restrictions.</p> <p>Secure multiparty computation and related privacy-enhancing technologies "show real promise" but are currently being applied only in narrow use cases due to complexity, performance overhead and integration challenges, Wells said.</p> </section> <section class="section main-article-chapter" data-menu-title="Consistency and flexibility in sovereignty"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Consistency and flexibility in sovereignty</h2> <p>Sovereignty programs aim to meet local requirements. But can they also achieve some level of consistency regarding policies, metadata, access control and quality checks -- even when data must remain in separate regions?</p> <p>"The effective pattern is centralized governance with local enforcement," Wells said.</p> <p>That is, organizations <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">standardize governance policies,</a> metadata definitions and control frameworks globally, then enforce them locally within each jurisdiction to respect sovereignty constraints, she noted.</p> <p>Wells described centralized metadata and governance as the enabling layer. Here, business glossaries, lineage management and defined data stewardship roles give teams visibility into where data resides, how it's used and which policies apply, regardless of location, she said. <br><br>"The goal is not to eliminate fragmentation -- it's to orchestrate it," Wells said.</p> <p>As a result, organizations maintain control while preserving the operational flexibility needed to run analytics and <a href="https://www.techtarget.com/searchcio/feature/How-CIOs-can-beat-AI-challenges-A-top-researchers-view">AI at scale</a>, she said.</p> <p>Given the changing regulatory landscape and evolving technologies, adaptability is a requirement for sovereignty.</p> <p>"It's the optionality that is really key here," Wolfe said. "Every cloud and AI decision already has sovereignty implications. The advantage goes to leaders who make those decisions deliberately, instead of making decisions by inaction."<br><br><i>John Moore is a freelance writer who has covered business and technology topics for 40 years. He focuses on enterprise IT strategy, AI adoption, data management and partner ecosystems.</i></p> </section> Geopolitical conflict and outages can upend assumptions about where data is controlled and accessed, pushing leaders to plan for jurisdictional risk, resilience and recovery. https://cdn.ttgtmedia.com/rms/onlineimages/strategy_a296343799.jpg https://www.techtarget.com/searchdatamanagement/feature/Data-sovereignty-expands-beyond-compliance-boundaries Fri, 15 May 2026 14:02:00 GMT <p>Power, not compute capacity, is becoming the primary constraint on where AI workloads and supporting platforms run -- and when they can go live.</p> <p>For enterprise data and analytics leaders, power concerns are no longer confined to data center facility planning. They are now a data architecture issue affecting AI deployments, platform scaling and resilience plans.</p> <p>Global electricity demand from data centers is projected to more than double by 2030, according to the International Energy Agency. Gartner forecasts AI workloads will use 44% of data center power by then. While the demand for power is surging due to AI, grid interconnection delays in key markets limit the availability of new data center capacity.</p> <p>Together, those forces are creating a new reality of power-constrained data architecture, where time-to-power becomes a core factor in AI infrastructure decisions alongside cost, latency and performance. The question is no longer just where infrastructure is cheapest or fastest, but <a href="https://www.techtarget.com/searchenterpriseai/feature/Is-GenAI-villain-and-hero-in-data-center-power-drama">where AI workloads can be powered</a>, scaled and protected from disruptions.</p> <p>Interconnection queues, equipment lead times and constrained regional capacity mean that power access is no longer a given but a planning variable. Organizations that do not account for it early in site selection, workload design and procurement face project delays, cost overruns and operational outages they could have anticipated. The impact is most visible in infrastructure planning, where power delivery timelines are beginning to shape data architecture decisions.</p> <section class="section main-article-chapter" data-menu-title="Time-to-power reshapes architecture and procurement"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Time-to-power reshapes architecture and procurement</h2> <p>The timeline from site selection to an energized data center facility has pushed well beyond typical project assumptions, and the delays are coming from multiple directions at once. An interconnection agreement does not necessarily mean power will arrive soon, even if data platforms are ready to deploy.</p> <p>Dell'Oro Group's February 2026 forecast projects the worldwide data center physical infrastructure market to grow by a mid-teens percentage each year through 2030, surpassing $80 billion by 2030. However, the market research firm said power scarcity is driving increased demand for on-site generation as data center operators work around grid constraints. On-site approaches are shifting from a contingency to a practical necessity for large facilities, it noted.</p> <p>Interconnection delays for power plants are also affecting data center timelines. A December 2025 report from Lawrence Berkeley National Laboratory found that power generation and storage capacity projects built in the U.S. from 2018 to 2024 took a median of more than four years to go from the interconnection request to commercial operation. In Northern Virginia, the country's largest data center market, transmission expansion is now a multiyear planning issue, extending to eight years or more. Timelines are comparable in parts of Europe, including the U.K. and Germany.</p> <p>The <a href="https://www.techtarget.com/searchenterpriseai/feature/Communities-call-for-transparency-in-AI-data-center-deals">pressure is not limited to hyperscalers</a> and large colocation providers that are securing hundreds of megawatts for AI deployments. Alex Cordovil, research director at Dell'Oro Group, said enterprise teams are already seeing downstream effects.</p> <p>"When hyperscalers and colocators absorb the lion's share of available utility capacity in a given market, enterprise teams face a knock-on effect, whether that's longer lead times for their own on-premises expansions, reduced availability at their preferred colocation facilities, or less favorable contract terms," Cordovil said.</p> <p>Energization dates -- when a facility can draw grid power -- now carry as much weight in site selection as network latency or real estate cost. Bloom Energy's 2025 data center power survey found that 84% of respondents ranked power access among their top three site-selection considerations. When timelines slip, organizations might need to move AI workloads to other regions, delay deployments or accept limited power access with potential interruptions during peak demand.</p> </section> <section class="section main-article-chapter" data-menu-title="The transition to on-site power and grid-interactive operations"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The transition to on-site power and grid-interactive operations</h2> <p>In addition to on-site power infrastructure they control directly, data center operators are turning to grid-interactive operations that adjust electricity use based on overall demand, reflecting a shift toward power-aware data platform design.</p> <p>Gartner expects energy alternatives, such as green hydrogen and <a href="https://www.techtarget.com/sustainability/feature/EmTech-MIT-2025-looks-at-the-future-of-US-nuclear-energy">small modular nuclear reactors</a>, to become more viable for data center microgrids by the end of the decade. Dell'Oro Group's quarterly tracking shows that worldwide data center physical infrastructure spending increased 20% year-over-year in the fourth quarter of 2025. Power distribution was one of the stronger growth areas, driven by demand for higher-density AI infrastructure.</p> <p>The shift is playing out across five technology categories:</p> <ul class="default-list"> <li><b>Strategic siting.</b> Colocating capacity near existing grid-connected infrastructure can help shorten deployment timelines.</li> <li><b>Microgrids.</b> Combining on-site generation with battery storage, microgrids enable facilities to operate independently of the grid during stress events to reduce the exposure of critical data workloads to instability.</li> <li><b>Fuel cells</b>. Solid-oxide fuel cells offer a faster path to primary power than traditional grid connections, deploying in weeks rather than years. They also run at significantly higher electrical efficiency than diesel generators, but buyers should assess cost, emissions reporting and runtime assumptions.</li> <li><b>Battery energy storage systems. </b>BESS can come online during outages, reducing dependence on diesel generators and supporting grid independence when a utility's power supply is interrupted.</li> <li><b>Workload flexibility.</b> AI and data workloads are increasingly designed to shed or shift load during peak grid stress without taking systems offline. Training workloads that tolerate latency are especially suited to this model.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Provider questions for power-constrained AI workloads"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Provider questions for power-constrained AI workloads</h2> <p>Understanding what data center operators are deploying is only part of the <a target="_blank" href="https://www.belfercenter.org/research-analysis/ai-data-centers-us-electric-grid" rel="noopener">equation</a>. Data leaders and other enterprise buyers also need to know whether the cloud and data platform providers they are evaluating are keeping pace.</p> <p>Cordovil said most buyers are still evaluating providers on the wrong metrics. Power usage effectiveness, redundancy tier and price per kilowatt are baseline measures. The questions that matter now, he added, are about power pathway resilience, future-proofing and flexibility:</p> <ul class="default-list"> <li>What is the provider's contracted utility capacity versus what is currently energized?</li> <li>How much headroom exists for <a href="https://www.techtarget.com/searchdatacenter/tip/Data-center-trends-to-watch">density growth</a>, not just at the facility level but at the rack and row level?</li> <li>What is the provider's strategy for on-site generation, grid interactivity and demand response participation?</li> <li>How flexible is the infrastructure to accommodate shifting power and cooling profiles as AI architectures evolve? Can it scale incrementally, or does it require a full replacement?</li> <li>What does the power roadmap look like over the next five years, not just what is available today?</li> </ul> <p>Tony Harvey, an analyst at Gartner, said teams should also ask whether promised power delivery is firm or can be reduced during grid stress events. If it's the latter, they should ask what conditions would trigger a reduction, how often it could happen and whether critical workloads can be protected from curtailment. If on-site generation is the backup plan, buyers should confirm whether it can carry the full power load and for how long.</p> </section> <section class="section main-article-chapter" data-menu-title="Planning ahead for power grid risk"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Planning ahead for power grid risk</h2> <p>Even with better provider selection, power risk remains a resilience issue for enterprise data teams. Power remains the leading cause of impactful data center incidents, according to Uptime Institute's annual outage analysis report for 2025. On-site generation and workload flexibility reduce exposure but do not eliminate risk.</p> <p>A complete resilience posture requires action across three areas:</p> <ul class="default-list"> <li><b>Physical separation across regions is the foundation.</b> Logical separation within a single facility provides no protection against a shared power failure. The baseline is cross-region replication of applications, data and logs across sites on separate feeds from separate providers, with failover tested under load rather than on paper.</li> <li><b>Generator and UPS testing must reflect actual failure scenarios.</b> A quarterly exercise that never reaches full load does not validate performance. Best practices include full load-bank testing quarterly; an annual black-start test where generators start cold and an uninterruptible power supply (UPS) bridges the gap; and verification that security systems and access controls hold during switchovers. Uptime Institute found failure to follow established procedures is the leading addressable cause of outages, making process review more impactful than new hardware.</li> <li><b>Workload classification is a prerequisite for everything else.</b> Teams that have mapped AI workloads by delay tolerance can negotiate demand-response agreements with utilities. That classification also provides a plan for shifting workloads across regions and power providers when a primary site faces constraints.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Terms should reflect power availability risk"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Terms should reflect power availability risk</h2> <p>AI workloads are increasingly moving closer to enterprise infrastructure for model fine-tuning, inference at the edge and retrieval-augmented generation, and the data architectures underpinning them are evolving quickly. The power and cooling envelope an organization needs to support platforms and applications today might look very different in 18 months.</p> <p>Power risk spans site selection, procurement and operations, but the contract with a provider defines an organization's exposure. Harvey said many buyers don't realize the terms are negotiable.</p> <p>He recommended that they:</p> <ul class="default-list"> <li>Require proof of a signed interconnection agreement, completed required studies and a confirmed energization date.</li> <li>Define delivery requirements at the rack power distribution unit level, including redundant feeds and completed full-load testing.</li> <li>Make the colocation provider responsible for utility delays where possible.</li> <li><a href="https://www.techtarget.com/searchsecurity/tip/What-CISOs-should-look-for-in-cyber-insurance-policy-fine-print">Review the force majeure clause</a> and explicitly exclude utility delays from its scope.</li> <li>Attach penalties to non-delivery tied to documented business impact.</li> <li>Require zero curtailment for critical loads backed by a service-level agreement with financial penalties.</li> </ul> <p>Getting the terms right is necessary, but it is only the starting point for data leaders. The broader risk of treating power as an afterthought is strategic, not just operational. <br><br>"Buyers who treat power as a static line item on a colocation contract are going to find themselves either capacity-constrained or locked into infrastructure that can't adapt as AI workloads scale and shift," Cordovil said.</p> <p><em>Sean Michael Kerner is an IT consultant, technology enthusiast and tinkerer. He has pulled Token Ring, configured NetWare and been known to compile his own Linux kernel. He consults with industry and media organizations on technology issues.</em></p> </section> Rising power demands and grid interconnection delays are hampering enterprise AI efforts and altering data strategies, workload placement and resilience planning. https://cdn.ttgtmedia.com/rms/onlineimages/location_g580502719.jpg https://www.techtarget.com/searchdatamanagement/feature/Power-constrained-data-architecture-curbing-AI-ambitions Wed, 13 May 2026 12:12:00 GMT <p>Vector databases have moved from a niche component of generative AI infrastructure to standard tooling across data-intensive industries. The technology now supports a range of enterprise applications across industries, with use cases primarily found where similarity-based retrieval outperforms traditional database queries.</p> <p><a href="https://www.techtarget.com/searchdatamanagement/tip/Vector-vs-graph-vs-relational-database-Which-to-choose">Unlike traditional relational databases</a> that store data in tables with rows and columns, vector databases represent unstructured data -- text, images, audio -- as points in a multidimensional space, then retrieve results by similarity rather than exact matching. Algorithms such as approximate nearest neighbor make this practical at scale, which is why the approach has become foundational to generative AI applications.</p> <p>The following 10 use cases for vector databases illustrate how vector databases are delivering value across industries today.</p> <section class="section main-article-chapter" data-menu-title="1. Natural language processing"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Natural language processing</h2> <p>Vector databases capture complex semantic relationships that traditional data models miss, <a href="https://www.techtarget.com/searchdatamanagement/opinion/Why-data-semantics-matters-for-context-aware-systems">making them well-suited</a> to language tasks such as sentiment analysis and translation. Embedding-based representations encode context and tone in ways dictionary lookups can't, supporting more accurate classification, search and machine translation.</p> <p>The meaning of words depends greatly on context and cultural nuances. Embeddings recognize that "buy," "purchase" and "acquire" share meaning while differing slightly in context. Phrases such as "splurged on" carry sentiment beyond their literal definitions that lexical matching misses entirely.</p> </section> <section class="section main-article-chapter" data-menu-title="2. Customer support"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Customer support</h2> <p>People who have experienced long waits with a pressing issue on a customer service call know that sentiment analysis is a critical skill for human customer service representatives. For online customer service, chatbots now feel a <a href="https://www.techtarget.com/searchcustomerexperience/feature/In-2023-generative-AI-made-inroads-in-customer-service">little closer to human interaction</a>. A knowledge base using vectors can analyze and respond to customer support tickets by accurately categorizing issues. The bot can also note underlying sentiment using techniques similar to natural language processing. Responses and prioritization are handled more satisfactorily for customers.</p> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> [Vector databases] are reshaping how people interact with and benefit from the vast amounts of data generated in the contemporary world. </figure> <i class="icon" data-icon="z"></i> </div> </blockquote> </section> <section class="section main-article-chapter" data-menu-title="3. Image and video recognition"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Image and video recognition</h2> <p>Vector databases convert the pixel data of images and videos into vector representations. Vector representations of images and video enable similarity-based search across visual content such as facial recognition, object detection and scene understanding with high accuracy.</p> <p>For example, an e-commerce site can tag products in images to enable shoppers to perform visual searches. A social media platform can also use the technique for content moderation, detecting policy violations in visual content to quickly flag explicit content in videos and images at scale without needing human case-by-case review.</p> </section> <section class="section main-article-chapter" data-menu-title="4. Financial services fraud detection"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Financial services fraud detection</h2> <p>Representing transactions as vectors across many dimensions -- time, location, amount and the relationship between transactions -- makes it possible to detect subtle, nonlinear patterns typical of fraudulent behavior. Banks can identify anomalous vectors with more nuance than traditional rules-based fraud models, catching novel patterns that fixed rules miss. These new capabilities result in fewer false declines on legitimate transactions and stronger fraud detection for banks and card issuers.</p> </section> <section class="section main-article-chapter" data-menu-title="5. E-commerce product recommendations"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. E-commerce product recommendations</h2> <p>E-commerce sites already deliver personalized product recommendations by analyzing customer browsing and purchase history. Data mining can identify correlations, but is too slow to respond to customer behavior on the website in near real time. Vector representations let recommendation systems match a shopper's current activity with similar products, customers and past sessions much faster, enabling e-commerce sites to offer a more engaging and insightful experience.</p> </section> <section class="section main-article-chapter" data-menu-title="6. Autonomous vehicles"> <h2 class="section-title"><i class="icon" data-icon="1"></i>6. Autonomous vehicles</h2> <p>Autonomous vehicle developers use vector databases in development workflows rather than the driving loop itself. Sensor logs from cameras, lidar, and radar captured across millions of fleet miles are encoded as vectors and stored for similarity search, allowing analysts to retrieve scenes resembling specific patterns <a href="https://www.techtarget.com/sustainability/feature/Breaking-the-cycle-of-algorithmic-bias-in-AI-systems">such as pedestrians</a>, traffic signals and obstacles.</p> </section> <section class="section main-article-chapter" data-menu-title="7. Medical diagnostics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>7. Medical diagnostics</h2> <p>The same approach applies to medical imaging. Scans such as MRIs and X-rays are encoded as vectors and compared against large data sets of cases with known conditions. The system can reveal patterns imperceptible to humans and help make an accurate diagnosis. As a result, doctors could accurately identify early indicators of cancer or other serious illnesses without invasive tests.</p> </section> <section class="section main-article-chapter" data-menu-title="8. Biometrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>8. Biometrics</h2> <p>Anyone who has recently traveled through an airport, applied for a driver's license or opened a bank account in person is familiar with biometric security. Vector analysis excels at biometric pattern recognition from fingerprints, facial characteristics and other identity attributes captured digitally for authentication. It converts slight but distinctive qualities to vectors. The result is a system that is much less likely to be confused by cuts on a person's fingers, haircuts, makeup or natural aging.</p> </section> <section class="section main-article-chapter" data-menu-title="9. Media recommendations"> <h2 class="section-title"><i class="icon" data-icon="1"></i>9. Media recommendations</h2> <p>Entertainment services, such as movie or music <a target="_blank" href="https://www.moltencloud.com/blog/film-industry/how-streaming-platforms-are-using-artificial-intelligence-in-2024" rel="noopener">streaming platforms</a>, make recommendations much like e-commerce sites, but with some important differences. Entertainment users often consume media in longer sessions -- such as watching a movie or binge-watching a series -- than online shoppers do. Instead of simple variables such as price and shipping options, recommendation systems use temporal dynamics such as new releases, viral content and even the mood of the viewer or listener. The process may be similar in principle, but in practice, vector databases are now essential for the complexity of media recommendations.</p> </section> <section class="section main-article-chapter" data-menu-title="10. Video games"> <h2 class="section-title"><i class="icon" data-icon="1"></i>10. Video games</h2> <p>In game development, vector databases support work that happens outside the real-time rendering loop. Studios use them to search large asset libraries by similarity, analyze player behavior telemetry for matchmaking and game balance, and, more recently, experiment with <a href="https://arxiv.org/html/2504.13928v1">LLM-driven character dialogue</a> that retrieves contextually relevant responses to player input.</p> <p>Vector databases are not just a technological innovation; they are reshaping how people interact with and benefit from the vast amounts of data generated in the contemporary world. Business applications are only scratching the surface of these models' potential. The flexibility and scale of vector databases offer intriguing possibilities for adaptive, engaging and analytically powered experiences in the future.</p> <p><b>Editor's note:</b> <i>This article was originally published in April 2024 and updated in May 2026 to reflect changes in how vector databases are used in production. </i></p> <p><em>Donald Farmer is a data strategist with 30+ years of experience, including as a product team leader at Microsoft and Qlik. He advises global clients on data, analytics, AI and innovation strategy, with expertise spanning from tech giants to startups. He lives in an experimental woodland home near Seattle.</em></p> </section> Vector databases have become the standard infrastructure for enterprise AI work. These 10 use cases for vector databases show where the technology delivers practical value. https://cdn.ttgtmedia.com/rms/onlineimages/iot_g1156740131.jpg https://www.techtarget.com/searchdatamanagement/tip/Top-industry-use-cases-for-vector-databases Mon, 11 May 2026 14:05:00 GMT <p>No data governance strategy is complete without a plan for tracking relevant KPIs and other metrics.</p> <p>Data governance metrics enable organizations to measure the effectiveness of governance processes across areas such as data quality, security, availability and usage. Collecting and analyzing metrics documents the <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">value of a data governance program</a>, helping data leaders and governance managers build an ongoing business case for it. If the program doesn't achieve desired outcomes, metrics highlight issues that must be addressed.</p> <p>Tracking governance metrics is also crucial for regulatory compliance. Although <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know">data protection and privacy regulations</a> typically don't include explicit rules on collecting governance metrics, many require organizations to demonstrate that they take reasonable measures to maintain data security, privacy and, in some cases, quality. Metrics in these areas help show regulators and compliance auditors that a company is acting responsibly.</p> <p>Organizations track different sets of metrics based on their governance goals and requirements. However, they commonly incorporate these six types of metrics to get a comprehensive view of their data governance program.</p> <section class="section main-article-chapter" data-menu-title="1. Operational metrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Operational metrics</h2> <p>The operational component of data governance centers on how an organization <a href="https://www.techtarget.com/searchdatamanagement/tip/6-key-steps-to-develop-a-data-governance-strategy">designs and implements its governance strategy</a>. The following are examples of metrics that provide insights into data governance operations:</p> <ul class="default-list"> <li><b>Total number of policies.</b> Tracking the number of data governance policies an organization has in place helps data leaders and business stakeholders understand the scope of governance operations and ensure that required resources are available to manage them.</li> <li><b>Governance assessment frequency and results.</b> Organizations should assess governance policies and procedures at fixed intervals, ideally once or twice per year. More frequent assessments might be required if data governance needs change quickly or if an organization undergoes a technological transformation -- such as increased AI adoption -- that has <a href="https://www.techtarget.com/searchdatamanagement/feature/AI-data-governance-guidance-that-gets-you-to-the-finish-line">major ramifications for governance policies</a>. Governance problems identified by other metrics can also trigger out-of-cycle assessments.</li> <li><b>Total data assets and data volumes.</b> Documenting the number of distinct data assets an organization has and the size of data volumes in different systems establishes resource baselines for data governance initiatives. Tracking changes in data assets and volumes also enables organizations to quickly scale governance operations when necessary.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="2. Data quality metrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Data quality metrics</h2> <p>Low-quality data makes it hard to manage business operations effectively and generate accurate insights in analytics and AI applications to enable better decision-making. Data quality metrics <a href="https://www.techtarget.com/searchdatamanagement/tip/Evaluating-data-quality-requires-clear-and-measurable-KPIs">validate data reliability</a> for planned uses and highlight issues for data management and governance teams to address.</p> <p>Different metrics <a href="https://www.techtarget.com/searchdatamanagement/tip/6-dimensions-of-data-quality-boost-data-performance">measure data quality attributes</a> such as accuracy, completeness, consistency, timeliness and validity. The following are some commonly used ones:</p> <ul class="default-list"> <li><b>Number of duplicate data entries.</b> Duplicate entries in data sets lead to misleading analytics results and cause problems in business operations such as marketing, customer service and logistics.</li> <li><b>Data error rate.</b> Data errors such as corrupted data, typos and transposed numbers are another source of analytics and operational problems. Acceptable error rates often vary for different applications.</li> <li><b>Percentage of data records with missing or incomplete values.</b> Data that lacks required information due to missing values or incomplete fields also affects applications.</li> <li><b>Transformation error rate.</b> Tracking how often data transformations fail helps organizations identify resulting data quality issues, such as corrupted, inaccurate or incomplete data sets, as well as opportunities to improve <a href="https://www.techtarget.com/searchdatamanagement/tip/The-difference-between-data-cleansing-and-data-transformation">data transformation processes</a>.</li> <li><b>Number of data issues identified and corrected.</b> Tracking the number of data issues found and fixed within a specified period helps document the progress of <a href="https://www.techtarget.com/searchdatamanagement/feature/Proactive-practices-for-data-quality-improvement">data quality improvement efforts</a>.</li> </ul> <figure class="main-article-image full-col" data-img-fullsize="https://www.techtarget.com/rms/onlineImages/data_management-common_data_quality_metrics-f.png"> <img data-src="https://www.techtarget.com/rms/onlineImages/data_management-common_data_quality_metrics-f_mobile.png" class="lazy" data-srcset="https://www.techtarget.com/rms/onlineImages/data_management-common_data_quality_metrics-f_mobile.png 960w,https://www.techtarget.com/rms/onlineImages/data_management-common_data_quality_metrics-f.png 1280w" alt="Graphic that lists commonly used data quality metrics." height="288" width="559"> <figcaption> <i class="icon pictures" data-icon="z"></i>Data management and governance teams use these metrics to assess data quality. </figcaption> <div class="main-article-image-enlarge"> <i class="icon" data-icon="w"></i> </div> </figure> </section> <section class="section main-article-chapter" data-menu-title="3. Data availability and usage metrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Data availability and usage metrics</h2> <p>Even high-quality data is only useful if business users and analytics teams can find and access it. Data availability and usage metrics provide insight into how effectively an organization uses data. Tracking them helps data management and governance teams identify technical issues in data platforms and problems with data discovery and accessibility.</p> <p>Examples of availability and usage metrics include the following:</p> <ul class="default-list"> <li><b>System uptime percentage.</b> Monitoring the uptime percentage of data platforms and analytics systems shows how often issues such as technical disruptions and cyberattacks interrupt data availability. This metric helps teams evaluate whether system changes or upgrades are required to prevent unexpected downtime.</li> <li><b>Average data latency.</b> Data latency measures how long it takes <a href="https://www.techtarget.com/searchbusinessanalytics/news/365534255/Data-pipelines-deliver-the-fuel-for-data-science-analytics">processing workflows and pipelines</a> to make data available for use after it's generated or updated. Tracking average latency shows whether initial availability goals -- often measured in seconds -- are being met.</li> <li><b>Data access frequency.</b> Measuring how often data is accessed helps determine whether its usage meets expected levels. Underutilization could indicate that data is difficult to access or that users aren't aware of available data resources due to inadequate training or <a href="https://www.techtarget.com/searchdatamanagement/answer/What-steps-are-key-to-building-a-data-catalog">shortcomings in a data catalog</a>. It could also be a sign that a data set isn't relevant or useful to many people.</li> <li><b>Data update frequency.</b> Monitoring the number of updates and modifications to data also provides insights into its usage. Data writes represent a more active form of engagement than reads. Consequently, data sets with a high update frequency might require closer oversight from data quality analysts and governance teams.</li> <li><b>Number of unique users.</b> Tracking the number of distinct users accessing data also helps identify underutilized data assets and guide data management and governance activities.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="4. Data security and privacy metrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Data security and privacy metrics</h2> <p>Effective data governance protects sensitive information from exposure or misuse that could create regulatory compliance issues and cause reputational damage. Data security and privacy metrics provide insights into the effectiveness of data governance policies and procedures and enable governance teams to proactively <a href="https://www.techtarget.com/searchdatamanagement/feature/Top-3-data-privacy-challenges-and-how-to-address-them">identify and address risks</a>.</p> <p>The following are some helpful data security and privacy metrics:</p> <ul class="default-list"> <li><b>Number of access violations.</b> Tracking how frequently unauthorized users attempt to access data helps teams evaluate the effectiveness of access controls and <a href="https://www.techtarget.com/searchsecurity/feature/How-to-create-a-data-security-policy-with-template">data security policies</a>. Unauthorized users include external attackers, employees with no legitimate need to access specific data sets or contractors who shouldn't have access to a company's internal data.</li> <li><b>Number of data breaches.</b> This is a separate metric for tracking incidents where unauthorized users <a href="https://www.techtarget.com/searchsecurity/tip/How-to-prevent-a-data-breach-10-best-practices-and-tactics">successfully access a company's data</a>.</li> <li><b>Number of privacy incidents.</b> This metric tracks improper access, use and handling of personal data that violates privacy laws and internal policies. In addition to data breaches involving personal information, privacy incidents include actions such as sharing data with unauthorized individuals and using it inappropriately in analytics and AI applications.</li> <li><b>Policy compliance rate.</b> This measures the percentage of data assets and processes that comply with data security and privacy policies, including data access controls, encryption standards, retention schedules and regulatory requirements.</li> <li><b>Number of policy exceptions.</b> Governance teams often grant temporary exceptions to data security and privacy policies. Tracking the number of exceptions helps manage them and identify areas where policies need to be updated. Teams can also track longer-term policy exemptions and compare the two numbers with the policy compliance rate to gain broader insights into data that isn't fully protected due to inadequate or unenforced policies.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="5. Data stewardship metrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. Data stewardship metrics</h2> <p>Data stewardship metrics focus on how effectively an organization engages with data governance practices and maintains governance standards. They track the <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-steward-responsibilities-fill-data-quality-role">activities of data stewards</a> who are directly responsible for managing data sets and overseeing the implementation of governance policies.</p> <p>Common data stewardship metrics include the following:</p> <ul class="default-list"> <li><b>Number of data stewards.</b> Tracking the number of data stewards assigned to data assets and comparing it to the number of assets and data volumes helps governance teams assess whether their organization has enough stewards -- or too many.</li> <li><b>Number of issues resolved.</b> The number of data governance problems that data stewards resolve is routinely tracked to monitor their activities and assess their productivity.</li> <li><b>Issue resolution time.</b> Similarly, tracking resolution times for data issues provides useful insights into data stewardship work and performance.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="6. Data literacy metrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>6. Data literacy metrics</h2> <p>Employees who understand the value of data and how to use it effectively are more likely to recognize the <a href="https://www.techtarget.com/searchdatamanagement/tip/Successful-data-operations-follow-a-data-governance-roadmap">need for strong data governance</a>. Organizations use data literacy metrics to assess literacy levels and how actively users participate in educational processes designed to <a href="https://www.techtarget.com/searchbusinessanalytics/feature/How-business-leaders-can-make-a-data-literate-culture-stick">create a more data-literate culture</a> that helps sustain governance initiatives.</p> <p>The following are examples of literacy metrics:</p> <ul class="default-list"> <li><b>Training engagement and completion rates.</b> Metrics that track user participation in <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Develop-a-data-literacy-program-to-fit-your-company-needs">data literacy training programs</a> and broader data governance education -- such as the percentages of employees who enroll in and successfully complete training programs -- offer insight into engagement with governance initiatives. The percentage involved in governance activities is another useful metric.</li> <li><b>Data literacy scores.</b> The average scores from data literacy assessments conducted during training programs measure how well employees understand the material presented and their skills in areas such as data discovery, analysis and visualization. Tracking them helps governance teams assess both data literacy levels and the effectiveness of the training.</li> <li><b>Internal survey results.</b> User surveys conducted by governance teams provide both quantitative and qualitative metrics on data literacy and awareness in organizations.</li> </ul> <p><em>Chris Tozzi is a freelance writer, research adviser, and professor of IT and society who has previously worked as a journalist and Linux systems administrator.</em></p> </section> Implementing a data governance program isn't enough. Data leaders also need to track and analyze various metrics to evaluate its effectiveness and address shortcomings. https://cdn.ttgtmedia.com/visuals/searchCloudComputing/development/cloudcomputing_article_008.jpg https://www.techtarget.com/searchdatamanagement/feature/Data-governance-metrics-Data-quality-data-literacy-and-more Fri, 08 May 2026 14:05:00 GMT <p>Low-code and no-code tools and AI-powered vibe coding accelerate data-related development work, but unexpected fallout can put enterprise analytics and AI applications at risk.</p> <p>These development approaches help shorten the distance from idea to impact by expanding who can innovate with data. In addition to IT teams, application owners and other citizen developers <a href="https://www.techtarget.com/searchenterpriseai/tip/Citizen-developers-are-redefining-enterprise-AI-development">can take the initiative</a> and address unmet data needs themselves. Wishful thinking by business users becomes creative problem-solving. Work gets done faster. But the speed of technology-enabled citizen development is colliding with data pipeline reliability issues in the enterprise. The downside of <a href="https://www.techtarget.com/searchsecurity/tip/Vibe-coding-security-risks-and-how-to-mitigate-them">unfettered development</a> is that hidden integration layers and ad hoc data pipelines create visibility gaps for data management teams. Despite the best intentions, the results can prove costly.</p> <p>A <a href="https://www.fivetran.com/blog/the-enterprise-data-infrastructure-benchmark-report-2026">survey</a> of 500 senior data and technology leaders conducted in late 2025 by data management vendor Fivetran found that legacy and DIY pipelines using traditional ETL processes break 30% to 47% more often than fully managed <a href="https://www.techtarget.com/searchdatamanagement/definition/Extract-Load-Transform-ELT">ELT</a> systems, resulting in an average monthly downtime of 60 hours. It also found that data engineers spend 53% of their time on pipeline maintenance, taking time away from supporting new analytics and AI uses.</p> <p>Anjan Kundavaram, chief product officer at Fivetran, said low-code, no-code and vibe coding all contribute to the DIY nature of many data conduits <a href="https://www.techtarget.com/searchenterpriseai/feature/Democratization-of-AI-creates-benefits-and-challenges">that escape oversight.</a></p> <p>"There's a hidden data integration layer, and there are hidden data pipelines," he said.</p> <p>Lina Vaskelė, chief risk and security officer at aircraft maintenance and repair services provider FL Technics, underscored this risk as businesses increasingly adopt low-code, no-code and AI-powered development tools.</p> <p>"I think the main issue is losing some visibility about the data, and how the outputs are used later,"&nbsp;she said.</p> <section class="section main-article-chapter" data-menu-title="Pain points in the pipeline"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Pain points in the pipeline</h2> <p>Attention to edge cases ranks among the key considerations in making data pipelines reliable, Kundavaram said. Potential wrinkles include changes to data sources.</p> <p>"There's a lot of incidental complexity in the actual data source, like Salesforce, Workday or any [application]," he said. "Those applications are changing over time. They are not static."</p> <p>An unmanaged, DIY pipeline might not be prepared to handle such changes. For instance, if a field in a data source's API is renamed, a pipeline built to look for that field will stumble when it encounters a different name. As a result, the pipeline delivers incomplete data to the target system, which could be a <a href="https://www.techtarget.com/searchdatamanagement/tip/Assemble-the-layers-of-big-data-stack-architecture">dashboard, data lake or data warehouse</a> -- or, increasingly, a data lakehouse that combines elements of the latter two platforms.</p> <p>Changes on the target side or in data throughput pose similar risks. A pipeline designed to handle a predictable capacity will break when there's a sudden spike. A source or target change combined with a spike in data volume could introduce cascading failures.</p> <p>No-code and vibe-coded pipelines often fail to anticipate those variables, Kundavaram said.</p> <p>"Some core parts of the data infrastructure are going to be very brittle. It's going to create a lot more outages," he said.</p> <p>The <a href="https://www.techtarget.com/searchhrsoftware/feature/Enterprise-AI-enters-its-operational-phase">rise of AI systems</a> only increases the risk of ungoverned data workflows. Hastily built and potentially hidden pipelines can feed flawed data into AI models, reducing their accuracy and eroding decision quality and trust.</p> <p>"The focus is shifting from just model performance to the reliability and transparency of the entire data ecosystem," said&nbsp;Amir Kazmi, chief technology and growth officer at Ralliant, which makes test and measurement tools, industrial sensors and other precision instruments. "The issue isn't dramatic model corruption as much as it is gradual decision degradation, where small inconsistencies compound over time."</p> <p>Kazmi said pipelines that are fragmented or not fully observable make it harder to trace how outputs are generated, validate the inputs and detect model drift.</p> </section> <section class="section main-article-chapter" data-menu-title="Governing low-code, no-code and vibe coding endeavors"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Governing low-code, no-code and vibe coding endeavors</h2> <p>Dealing with this situation is a <a href="https://www.techtarget.com/searchdatamanagement/feature/AI-data-governance-guidance-that-gets-you-to-the-finish-line">matter of governance</a>, said Thomas Squeo, CTO for the Americas at Thoughtworks, a technology consulting firm. But governance mechanisms should recognize the benefits of tools that better enable citizen development.</p> <p>"I don't think the answer is to lock these tools down, because the value is outsized relative to the risk," Squeo said.</p> <p>Vaskelė said many employees outside of IT have a passion for exploring tools and using them to save time on daily tasks. Advising citizen developers on best practices works better than trying to control tool use, she added.</p> <p>"I think that shadow IT, or shadow systems, will exist because people will always build, will always experiment," she said. "The best approach is to guide them and to define the boundaries."</p> <p>At Lithuania-based FL Technics, policies determine those boundaries, while employee training provides guidance on proper tool use, Vaskelė said. Employees are required to use tools within the corporate account. New tools need IT approval and a vendor cybersecurity assessment.</p> <p>Jason Brucker, a managing director at consulting firm Protiviti, said low-code, no-code and AI platforms simplify data integration and support process modernization, but raise governance and maintainability concerns.&nbsp; &nbsp;</p> <p>As a result, some organizations might be tempted to prevent the use of low-code/no-code data integrations outside specific IT teams, Brucker noted. But doing so does not guarantee success and can lead users to circumvent internal policies and procedures, he said. Strong governance, paired with data awareness and training, is a better approach, Brucker said, adding that well-designed governance initiatives <a href="https://www.techtarget.com/searchdatamanagement/opinion/Most-data-governance-wasnt-built-for-AI">establish clear guidance</a> and include validation expectations and monitoring.</p> </section> <section class="section main-article-chapter" data-menu-title="Boosting visibility, sticking to the basics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Boosting visibility, sticking to the basics</h2> <p>The balancing act of managing these tools while preserving development speed calls for comprehensive monitoring, particularly of key data pipelines.</p> <p>"Make the invisible, visible," Vaskelė said.</p> <p>To that end, enterprises should map their processes and prioritize those most critical to the business, she said. For FL Technics, examples include workflows in aircraft maintenance hangars. Vaskelė said that, in such cases, data lineage is crucial for showing where data flows and identifying its original source.</p> <p>Visibility into new tools <a href="https://www.computerweekly.com/news/366640697/Why-OpenClaw-agents-are-the-next-big-enterprise-challenge">entering the market</a> at astonishing speed is also important.</p> <p>"It's almost like a biome where, all of a sudden, there's all this new life growing at every turn, and you don't know necessarily how you are going to control it," Squeo said.</p> <p>He said the goal is to be aware of the tools and categorize them: "These are the ones that are dangerous, these are the ones that are acceptable, and these are the ones that are unknown and need to be determined."</p> <p>That assessment is an ongoing activity for chief data officers and other leaders, Squeo added.</p> <p>On controls for these citizen development tools, Squeo emphasized the fundamentals: <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">zero-trust principles,</a> data loss prevention (DLP), and managing north-south and east-west traffic.</p> <p>"Those are just good practices to overlay, and that's a combination of network management and DLP with a good security posture," he said. "Verify that you are doing the basics."</p> <p>In addition, Squeo said some organizations deploy network proxies that provide baseline visibility into AI tool access and govern outbound data flows, especially when combined with DLP and network security policies. Proxies also help identify usage patterns, enforce access restrictions and <a href="https://www.techtarget.com/searchsecurity/tip/How-to-perform-a-data-risk-assessment-step-by-step">reduce obvious risk exposure.</a> However, he noted that because proxy servers operate primarily at the network layer, they lack visibility into the context, sensitivity or intent of data -- especially with encrypted traffic.</p> <p>Proxies improve detection but don't fully resolve <a href="https://www.techtarget.com/searchsecurity/tip/Shadow-code-The-hidden-threat-for-enterprise-IT">ungoverned data flows or shadow AI</a> architectures, Squeo said.</p> <p>Kazmi said Ralliant's approach to preventing issues with data pipelines and integration layers consists of three pillars:</p> <ul class="default-list"> <li><b>Ownership.</b> Treat critical data flows as products with accountable owners for quality, availability and use.</li> <li><b>Platform.</b> Standardize on governed environments with embedded validation, access controls and policies.</li> <li><b>Visibility. </b>Continuously monitor pipelines, understand dependencies and validate performance in real time.</li> </ul> <p>"But what organizations are learning, ourselves included, is that speed redistributes complexity," Kazmi said.</p> </section> <section class="section main-article-chapter" data-menu-title="Moving beyond vibe coding"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Moving beyond vibe coding</h2> <p>Yasmeen Ahmad, managing director of product management for Google Cloud's data and AI cloud platform, said low-code and no-code are firmly establishing themselves in the enterprise because AI and agent technologies remove some of the manual toil in data preparation and pipeline work.</p> <p>"This is one of the first use cases where we have seen almost instant ROI," she said, noting the historically labor-intensive work of software engineers, data practitioners and developers.</p> <p>"Think about data engineering pipelines: There are armies of humans across our largest customers building data pipelines," Ahmad said.</p> <p>Skilled data engineers, rather than business users, tend to adopt agentic AI tools to quickly vibe-code a data pipeline, she said. At Google, that is evolving toward intent-driven engineering: A data engineer states the intent of a pipeline, and Google's Gemini chatbot provides a plan. The engineer iterates on the plan, and the AI tool writes the code against the updated version.&nbsp;</p> <p><i>John Moore is a freelance writer who has covered business and technology topics for 40 years. He focuses on enterprise IT strategy, AI adoption, data management and partner ecosystems.</i></p> </section> Low-code/no-code and vibe coding require data leaders to shift from gatekeepers to architects of trust to maintain data integrity and keep pipelines running reliably. https://cdn.ttgtmedia.com/rms/onlineimages/code_g1127196618.jpg https://www.techtarget.com/searchdatamanagement/feature/Govern-citizen-development-to-avoid-data-pipeline-downtime Fri, 08 May 2026 09:53:00 GMT <p>Data governance tends to falter when policies exist, but no one owns the decisions behind them.</p> <p>Large enterprises, faced with <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know">increasing regulation</a> and heightened customer and investor scrutiny, have raised the stakes for enterprise decision-making. Implementing a data governance framework establishes a governance operating model, but few enterprises maintain a team that actually administers the program. Policy implementation is often left to individual departments or handed to IT teams with strong technical skills but limited business and regulatory knowledge, leaving gaps in compliance.</p> <p>As AI moves from pilot projects into production, companies steadily recognize that <a href="https://www.techtarget.com/searchenterpriseai/feature/9-data-quality-issues-that-can-sideline-AI-projects">data quality and availability</a> are binding constraints, as poorly governed data produces poorly governed AI systems. Shadow AI <a href="https://www.cybersecuritydive.com/news/shadow-ai-security-risks-netskope/808860/" target="_blank" rel="noopener">further complicates governance</a> by shifting decisions -- if they are made at all -- to employees without relevant experience, authority or training.</p> <p>A framework alone cannot adapt to these dynamics. It's a human problem. The question for senior leadership isn't whether to govern but <i>who</i> governs, how authority is structured and what business value the function delivers.</p> <p>The purpose of a data governance team isn't merely policy or regulatory compliance. The greater goal is enhancing the organization's capacity to <a href="https://www.techtarget.com/searchbusinessanalytics/feature/Data-driven-decision-making-case-study-Indeed">act with confidence using data-informed decisions</a>. Teams that accomplish this operate with both authority and responsibility, guided by an executive sponsor who treats the role as a standing commitment.</p> <section class="section main-article-chapter" data-menu-title="Executive sponsorship vs. data governance leadership"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Executive sponsorship vs. data governance leadership</h2> <p>Executive sponsorship is often treated as a single milestone: secure it. But that's only the first step. Trying to keep that sponsor engaged is harder and more important.</p> <p>Data leaders trying to secure sponsorship must determine whether the sponsor will commit time and influence on advancing the governance initiative. The <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-responsibilities-now-belong-in-the-C-suite">executive sponsor</a> must understand the job is more involved than initially expected. The division of labor between an executive sponsor and a data governance leader is as follows.</p> <p><iframe title="Executive sponsor vs. data governance leader" aria-label="Table" id="datawrapper-chart-TBHZJ" src="https://datawrapper.dwcdn.net/TBHZJ/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="713" data-external="1"></iframe></p> <p> <script type="text/javascript">window.addEventListener("message",function(a){if(void 0!==a.data["datawrapper-height"]){var e=document.querySelectorAll("iframe");for(var t in a.data["datawrapper-height"])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data["datawrapper-height"][t]+"px";r.style.height=d}}});</script> </p> </section> <section class="section main-article-chapter" data-menu-title="Functions and duties in the data governance model"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Functions and duties in the data governance model</h2> <p>The structure of a governance team depends on the vulnerabilities and risks it manages, not on a generic template. However, a compact set of roles recurs in mature programs.</p> <p><iframe title="Data governance team roles and responsibilities" aria-label="Table" id="datawrapper-chart-ceUwt" src="https://datawrapper.dwcdn.net/ceUwt/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="1014" data-external="1"></iframe> <script type="text/javascript">window.addEventListener("message",function(a){if(void 0!==a.data["datawrapper-height"]){var e=document.querySelectorAll("iframe");for(var t in a.data["datawrapper-height"])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data["datawrapper-height"][t]+"px";r.style.height=d}}});</script> </p> </section> <section class="section main-article-chapter" data-menu-title="Measuring the value of governance"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Measuring the value of governance</h2> <p>Governance rarely moves the needle on company performance directly, so measurement must be indirect but rigorous. Avoiding fines is not meaningful KPI; it is a baseline requirement. Useful metrics fall into three families.</p> <p><iframe title="The three families of data governance metrics" aria-label="Table" id="datawrapper-chart-T2K7K" src="https://datawrapper.dwcdn.net/T2K7K/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="501" data-external="1"></iframe></p> <p> <script type="text/javascript">window.addEventListener("message",function(a){if(void 0!==a.data["datawrapper-height"]){var e=document.querySelectorAll("iframe");for(var t in a.data["datawrapper-height"])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data["datawrapper-height"][t]+"px";r.style.height=d}}});</script> </p> <p>Other measures -- often called "vanity metrics" -- track activity rather than value. Examples include the number of policies authored, council meetings held or training hours delivered.</p> <p>Program effectiveness also depends on clear workflows. Three elements distinguish programs that work from those that struggle:</p> <ol class="default-list"> <li><b>Explicit dispute resolution paths.</b> When business units <a href="https://www.techtarget.com/searchdatamanagement/tip/The-data-ownership-blind-spots-putting-organizations-at-risk">disagree about data ownership</a> or definitions, someone must be named the decision-maker. Everyone should know who that is in advance.</li> <li><b>Documented decision rights.</b> Record who makes decisions, who is consulted and who is informed. A simple RACI matrix should prevent <a href="https://www.techtarget.com/searcherp/tip/RACI-matrix-for-project-management-success-with-example">much of the political friction</a> that governance programs generate. Ensure the RACI matrix is always up to date.</li> <li><b>Regular cadence.</b> Not every organization will have the same meeting rhythm, which should match its business pace. Monthly stewardship forums, quarterly team reviews and annual strategy resets work for most enterprises. But some organizations change more quickly and need a different cadence. &nbsp;Mergers and acquisitions also temporarily change cadence. In any instance, maintain transparent communication regarding decisions.</li> </ol> <p><em>Donald Farmer is a data strategist with 30+ years of experience, including as a product team leader at Microsoft and Qlik. He advises global clients on data, analytics, AI and innovation strategy, with expertise spanning from tech giants to startups. He lives in an experimental woodland home near Seattle.</em></p> </section> Regulations, AI and jumbled implementation oversight weaken decision-making. A dedicated team can help bring structure, accountability and consistent outcomes. https://cdn.ttgtmedia.com/visuals/searchCRM/learning_center/CRM_article_032.jpg https://www.techtarget.com/searchdatamanagement/feature/Who-belongs-on-a-high-performance-data-governance-team Wed, 06 May 2026 16:00:00 GMT <p>Without effective <a href="https://www.techtarget.com/searchdatamanagement/definition/data-governance">data governance</a>, growing volumes of data in IT systems are likely to become a disorganized morass, limiting their potential use. The risk of data misuse also increases due to lax controls.</p> <p>Conversely, well-governed data is consistent and accessible across the enterprise, enabling better-informed business decisions and <a href="https://www.techtarget.com/searchbusinessanalytics/opinion/Why-the-rush-to-replace-dashboards-with-AI-is-a-mistake">more accurate analytics and AI applications.</a> Companies are also less likely to experience serious data breaches or <a href="https://www.techtarget.com/searchdatamanagement/feature/Top-3-data-privacy-challenges-and-how-to-address-them">data privacy issues</a>, reducing their exposure to regulatory compliance problems, legal liabilities and reputational damage.</p> <p>As a result, developing a data governance strategy is a <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-responsibilities-now-belong-in-the-C-suite">high-priority item on the C-suite agenda</a> in well-run companies. The chief data officer and other data leaders play central roles in that process, typically managing it and working closely with their business counterparts to create and then implement the governance strategy.</p> <p>Getting started with data governance is a big undertaking that commonly requires a substantial budget and significant resource commitments. It might be tempting to buy a strategy from a consultancy or a <a href="https://www.techtarget.com/searchdatamanagement/feature/15-top-data-governance-tools-to-know-about">data governance software vendor</a> that promises a packaged set of policies and tools. But for optimal alignment with your organization's business operations and processes, it's best to develop a tailored one in-house, using these seven steps.</p> <section class="section main-article-chapter" data-menu-title="1. Document existing data governance processes"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Document existing data governance processes</h2> <p>Your company likely already has some data governance processes that should be incorporated into a formal strategy or replaced with new ones. Various people manage and oversee corporate data -- database administrators, backup admins, data architects and data quality analysts, for example. Document this by creating a directory of data assets and a corresponding list of managers and staff who are responsible or accountable for data.</p> <p>Don't be surprised if this exercise reveals some sobering, even shocking, oversights and gaps. The existing informal approach might reflect a messy reality, but getting a picture of current processes sets the stage for establishing a more strategic data governance program.</p> </section> <section class="section main-article-chapter" data-menu-title="2. Secure executive sponsorship for the data governance program"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Secure executive sponsorship for the data governance program</h2> <p>Enlist senior business executives to sponsor, fund and promote the governance program. Their buy-in and top-down influence are critical because effective data governance requires participation and cooperation by departments and business units across the enterprise. But how can you win executive support for an initiative that might not show clear bottom-line benefits, at least not right away?</p> <p>Invoking fear, uncertainty and doubt is the usual default method. Horror stories of inaccurate data leading to bad business decisions, or of fines for failing to comply with <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know">data privacy and protection laws</a>, might be enough to convince business leaders to back a governance initiative. By itself, however, this defensive approach isn't the optimal way to secure long-term data governance commitments.</p> <p>Instead, combine it with a more forward-looking approach. Explain that data governance is largely informal now and that the company needs a framework with more defined processes. Emphasize that implementing one will not only help <a href="https://www.techtarget.com/searchdatamanagement/feature/Proactive-practices-for-data-quality-improvement">improve data quality</a> and meet regulatory requirements but also make the organization more functional and resilient.</p> <p>Also, address upfront an issue that often causes resentment among business stakeholders and users: the perception that data governance stifles creative uses of data. Data governance policies don't restrict innovation. It's quite the opposite: By creating a more reliable data foundation, effective governance enables new ideas to flourish while reducing the risk of improper data use.</p> <figure class="main-article-image full-col" data-img-fullsize="https://www.techtarget.com/rms/onlineImages/data_management-need_to_govern_data-f.png"> <img data-src="https://www.techtarget.com/rms/onlineImages/data_management-need_to_govern_data-f_mobile.png" class="lazy" data-srcset="https://www.techtarget.com/rms/onlineImages/data_management-need_to_govern_data-f_mobile.png 960w,https://www.techtarget.com/rms/onlineImages/data_management-need_to_govern_data-f.png 1280w" alt="Visual that lists key reasons why organizations need a data governance program." height="283" width="560"> <figcaption> <i class="icon pictures" data-icon="z"></i>These are key reasons why organizations need to develop and implement an effective data governance strategy. </figcaption> <div class="main-article-image-enlarge"> <i class="icon" data-icon="w"></i> </div> </figure> </section> <section class="section main-article-chapter" data-menu-title="3. Improve data literacy and skills across the organization"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Improve data literacy and skills across the organization</h2> <p>End users who understand the potential value of data and how to use it effectively are more likely to recognize the need to protect data assets and prevent misuse. To foster that understanding across the enterprise, <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Develop-a-data-literacy-program-to-fit-your-company-needs">develop training to improve data literacy and skills</a> as part of the governance strategy.</p> <p>Enhanced data literacy also helps raise up data governance efforts in another way. End users often create duplicate reports, dashboards, spreadsheets and even entire databases because they don't know how to find existing ones. A <a href="https://www.techtarget.com/searchbusinessanalytics/feature/How-business-leaders-can-make-a-data-literate-culture-stick">data-literate culture</a> is better equipped to discover and reuse such assets, increasing efficiency and consistency and reducing the risk of data errors. This, in turn, helps streamline governance tasks.</p> </section> <section class="section main-article-chapter" data-menu-title="4. Create a virtual governance team at first, then formalize roles"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Create a virtual governance team at first, then formalize roles</h2> <p>It's too much to ask a company to reorganize upfront to improve data governance. Instead, start by <a href="https://www.techtarget.com/searchdatamanagement/feature/Who-belongs-on-a-high-performance-data-governance-team">constructing some organizational scaffolding</a> around the existing ad hoc data structures. Identify the key roles currently involved in governance processes and create a virtual team to improve coordination and collaboration.</p> <p>As governance becomes more formalized, new roles will emerge. A data governance manager or vice president commonly leads a team of data governance specialists who coordinate the program. Some business or IT workers will become data stewards, with direct responsibility for implementing governance policies in particular data sets. That can be a full-time or part-time role, depending on the organization's size and the complexity of its governance needs.</p> <p>Establishing a data governance council or committee is also a must. It typically includes the following members:</p> <ul class="default-list"> <li>Representatives from all departments and business units.</li> <li>IT, legal and compliance executives.</li> <li>Data stewards or others with data ownership responsibilities.</li> </ul> <p>The council sets data governance policies, creates common data standards, prioritizes governance projects and resolves data-related disputes, among other responsibilities. Having one ensures there's broad input on data governance controls and helps pave the way for enterprise-wide adoption of the governance strategy.</p> </section> <section class="section main-article-chapter" data-menu-title="5. Decide how to measure the governance program's effectiveness"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. Decide how to measure the governance program's effectiveness</h2> <p>For a data governance program to gain and maintain support in an organization, it's crucial to measure its effectiveness -- and show how it benefits the company. But effective data governance might not tangibly affect business performance. It also isn't easy to calculate KPIs for reduced business risks, such as avoiding regulatory fines or reputational damage.</p> <p>Instead, identify <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-metrics-Data-quality-data-literacy-and-more">key data governance metrics to track</a> and tie them to business benefits such as improved decision-making, optimized business processes and stronger privacy protections. For example, use metrics on data accuracy, completeness, consistency, timeliness and duplication to <a href="https://www.techtarget.com/searchdatamanagement/tip/6-dimensions-of-data-quality-boost-data-performance">monitor data quality levels and document improvements</a> that make data more reliable for analytics and AI applications.</p> <p>Metrics also help identify governance issues. Tracking how often users access data provides insight into whether it's being used effectively. Low usage might indicate a lack of awareness or accessibility. An increase in the overall number of analytics users is a marker of the governance program's success, but further training might be required if metrics also show new users are creating reports and dashboards that duplicate existing ones.</p> </section> <section class="section main-article-chapter" data-menu-title="6. Prioritize data governance for new AI applications"> <h2 class="section-title"><i class="icon" data-icon="1"></i>6. Prioritize data governance for new AI applications</h2> <p>Governing data for AI applications is now a key consideration for data leaders and governance teams. Data readiness is <a href="https://www.techtarget.com/searchdatamanagement/tip/Experts-share-practices-to-overcome-AI-data-readiness">critical to successful deployments</a> of machine learning, generative AI and agentic AI tools. Effective data governance ensures that AI models are built on a solid foundation of high-quality data and don't use it in ways that violate privacy and ethics policies.</p> <p>For example, retrieval-augmented generation (RAG) frameworks pose specific data governance challenges in enterprise AI applications. RAG enables large language models (LLMs) to <a href="https://www.techtarget.com/searchenterpriseai/opinion/How-RAG-unlocks-the-power-of-enterprise-data">directly draw from enterprise data</a> -- it retrieves relevant documents or records from internal knowledge bases and uses them to generate responses to user queries.</p> <p>From a governance perspective, successful RAG use requires not only accurate, up-to-date data but also <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-lineage-documentation-imperative-to-data-quality">data lineage documentation</a> that traces an LLM's output back to the original data sources for explainability and performance auditing. Well-managed access control is also necessary. Define and enforce user permissions in the RAG framework to prevent end users from inadvertently seeing data they can't access in conventional analytics applications.</p> <p>In addition to incorporating AI-related governance processes into your data governance strategy, <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-and-AI-governance-must-team-up-for-AI-to-succeed">align them with an AI governance program</a> that monitors and controls AI deployments more broadly. Data governance and AI governance are separate functions, but they go hand in hand and should be tightly integrated.</p> </section> <section class="section main-article-chapter" data-menu-title="7. Select technologies that fit the data governance strategy"> <h2 class="section-title"><i class="icon" data-icon="1"></i>7. Select technologies that fit the data governance strategy</h2> <p>Various technologies can be used in data governance initiatives. Data governance software automates program management tasks, such as policy development, process documentation, data classification and workflow management. Data catalogs provide a <a href="https://www.techtarget.com/searchdatamanagement/answer/What-steps-are-key-to-building-a-data-catalog">unified inventory of data assets</a>, with built-in governance, data lineage and data curation features. Analytics catalogs help users find relevant dashboards, reports and data&nbsp;visualizations and provide guidance on how to use them appropriately.</p> <p>For data processing, newer data lakehouse architectures combine the raw data storage of a data lake with the structured, governed repository of a data warehouse. Collapsing the separation between those two platforms streamlines data management and governance work and provides a single system that supports BI, advanced analytics and AI applications.</p> <p>But don't build a data governance strategy around specific technologies. Selecting tools that align with the strategy and support its goals will put the governance program on track to deliver the expected benefits, rather than running into a technology dead end that undermines the program.</p> </section> <section class="section main-article-chapter" data-menu-title="Move your organization forward with data governance"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Move your organization forward with data governance</h2> <p>The value of a pragmatic data governance strategy should steadily grow over time. A well-designed data governance process empowers business and analytics teams to do more with data, fostering a more data-driven and insightful organization. Rather than being seen as a restrictive discipline imposed on the organization, effective governance will be embraced as a strategic foundation that ensures data is treated as a critical business asset. Ultimately, it helps enhance decision-making, optimize business operations and enable the company to gain a competitive advantage over less data-centric rivals.</p> <p><b>Editor's note:</b> <i>This article was updated in April 2026 for timeliness and to add new information.</i></p> <p><em>Donald Farmer is a data strategist with 30-plus years of experience, including as a product team leader at Microsoft and Qlik. He advises global clients on data, analytics, AI and innovation strategy, with expertise spanning from tech giants to startups.</em></p> </section> A strong data governance strategy enables more effective data use and helps prevent financial, legal and reputational problems. Follow these steps to develop one. https://cdn.ttgtmedia.com/rms/onlineimages/legal_g1169668297.jpg https://www.techtarget.com/searchdatamanagement/tip/6-key-steps-to-develop-a-data-governance-strategy Fri, 24 Apr 2026 17:01:00 GMT <p>Even with a governance program in place, organizations can still fall short.</p> <p>Lack of data ownership is a widespread issue. Many organizations are finding that vast amounts of data lack an assigned owner. According to the <a target="_blank" href="https://cpl.thalesgroup.com/about-us/newsroom/ai-the-new-insider-threat-facing-organizations" rel="noopener">2026 Data Threat Report</a> from global technology company Thales, only 34% of organizations know where all their data is stored, and only 39% can fully classify their data.</p> <p>Executives must identify unowned datasets within their organizations and assign ownership to ensure enterprise data is appropriately governed and secured. Not doing so puts the organization at financial and reputational risk.</p> <section class="section main-article-chapter" data-menu-title="Common data ownership oversights"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Common data ownership oversights</h2> <p>Quality data has become a critical asset for organizations, especially for <a href="https://www.techtarget.com/searchenterpriseai/feature/9-data-quality-issues-that-can-sideline-AI-projects">automation, analytics and AI</a>. Data owners, data stewards and chief data officers are responsible for ensuring that data meets established standards and <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-lineage-documentation-imperative-to-data-quality">has trackable lineage</a>.</p> <p>But there are still types of data that organizations fail to capture and govern effectively, including:</p> <ul class="default-list"> <li><b>Unstructured content from communication channels.</b> Transcripts from messaging and collaboration apps, meeting transcripts and recordings, and content from emails and social media exchanges all lack ownership but might contain critical business information.</li> <li><b>Data generated by shadow IT systems.</b> About 70% of CIOs believe that the business units in their organizations deploy unsanctioned tech, according to Flexera's <a href="https://www.flexera.com/resources/reports/ITV-REPORT-IT-Priorities" target="_blank" rel="noopener">2026 IT Priorities Report</a>. That creates shadow data -- that is, data generated and stored outside the purview of the organization's IT and security controls as well as its data governance program.</li> <li><b>Data developed in sandboxes and for temporary projects.</b> This spans from developmental databases and test environments to intermediate datasets to <a href="https://www.techtarget.com/searchenterpriseai/tip/Explore-the-role-of-training-data-in-AI-and-machine-learning">training data for AI and machine learning</a> models.</li> <li><b>Siloed data.</b> Many data types fall into this category, including data stored in individual desktop folders, temporary reports, disconnected spreadsheets and data in legacy systems. Some also put vendor and third-party data in this category. Dark data also falls into this category, even though the business no longer uses it.</li> <li><b>Operational and machine data.</b> Various data types belong here, ranging from systems logs, IoT sensor data and API payloads to metadata. These datasets often lack a single owner and are overlooked in management and governance.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Unowned data governance challenges"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Unowned data governance challenges</h2> <p>Establishing governance and ownership over unowned data is challenging. Experts noted that these data types are hard to classify, easy to overlook and span multiple business functions. Consequently, no single business leader claims or is assigned ownership. Or it might be that data ownership lies with many or all leaders, which lacks accountability.</p> <p>Regardless, unowned data puts the organization at risk of costly data breaches, <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know">regulatory violations</a> and inaccurate outputs from analytics, automation and AI initiatives due to poor data quality.</p> <p>With such consequences in mind, many executives seek to improve their organization's overall data management and governance. According to Workvia's <a target="_blank" href="https://www.workiva.com/sites/workiva/files/pdfs/workiva-2026-exec-benchmark-survey-en.pdf" rel="noopener">2026 Executive Benchmark Survey</a>, business leaders ranked "strengthening data governance" as the second-highest priority for digital transformation projects, behind automating data collection and validation.</p> <p>Similarly, Informatica's <a href="https://www.informatica.com/about-us/news/news-releases/2026/01/20260127-new-global-cdo-report-reveals-data-governance-and-ai-literacy-as-key-accelerators-in-ai-adoption.html" target="_blank" rel="noopener">CDO Insights 2026 Report</a> found that 86% of 600 surveyed data leaders planned to increase data management investments -- and 41% seek to boost data and AI governance. This is in response to concerns about poor data quality affecting business objectives.</p> <p>Ultimately, data management and governance won't succeed if they can't be accounted for. Assigning ownership to unowned or over-shared data will help organizations strengthen data governance and boost data quality -- and their own trustworthiness.</p> <p><em>Mary K. Pratt is an award-winning freelance journalist with a focus on covering enterprise IT and cybersecurity management.</em></p> </section> Organizations can't claim to have good data governance when they still have unowned data. Assigning ownership to siloed and dark data is critical to enterprise success. https://cdn.ttgtmedia.com/rms/onlineimages/folder-files07.jpg https://www.techtarget.com/searchdatamanagement/tip/The-data-ownership-blind-spots-putting-organizations-at-risk Fri, 24 Apr 2026 14:20:00 GMT <p>Data is one of an organization's most valuable assets. But without a comprehensive data strategy as a foundation, it often becomes fragmented, inconsistent and difficult to access or trust for business decision-making.</p> <p>An effective enterprise data strategy establishes a structured approach to managing, governing and using data in alignment with business objectives. That enables companies to <a href="https://www.techtarget.com/searchdatamanagement/opinion/Turning-data-into-a-strategic-advantage">unlock greater value from their data assets</a> through improved decision-making, optimized business processes and increased operational efficiency. It also helps them boost innovation and gain a sustainable competitive advantage over less data-driven rivals.</p> <p>The data strategy shouldn't focus on implementing new technologies. That comes later, driven by the strategy. Instead, it should set the direction for data management processes, address common data-related challenges and build the capabilities needed to <a href="https://www.techtarget.com/searchdatamanagement/opinion/Data-management-and-governance-key-to-successful-AI-use">support planned data use</a> across the enterprise.</p> <p>Follow these 12 steps to develop a data strategy that accomplishes those things and positions your organization to realize long-term business benefits.</p> <section class="section main-article-chapter" data-menu-title="1. Define clear business objectives for data initiatives"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Define clear business objectives for data initiatives</h2> <p>A successful enterprise data strategy is grounded in close alignment between data initiatives and business goals. Data management and analytics efforts should directly support priorities such as enabling better-informed decision-making, enhancing customer experience, improving business operations, <a href="https://www.techtarget.com/searchdatamanagement/feature/Modern-data-architectures-as-a-risk-management-strategy">reducing risks</a> and fostering innovation.</p> <p>To achieve this alignment, work closely with senior executives and business managers to identify critical objectives that depend on effective data use. Engaging with key stakeholders at the outset ensures the data strategy addresses real business needs and guides appropriate technology choices to help meet them. Data initiatives tied to measurable business outcomes are more likely to gain executive support and sustained investment in the resources required for long-term success.</p> </section> <section class="section main-article-chapter" data-menu-title="2. Assess the existing data landscape in your organization"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Assess the existing data landscape in your organization</h2> <p>Next, get a complete understanding of the organization's current data environment. A comprehensive assessment documents existing technologies, capabilities, challenges and opportunities for improvement. The data management team should conduct it with clear visibility across data domains and business processes enterprise-wide.</p> <p>As part of the assessment, review source systems, data platforms, integration processes, governance structures and analytics applications, as well as how data flows between IT systems in different departments or business units. This uncovers issues such as data silos, inconsistent data definitions, limited metadata visibility and restricted access to relevant data. Identifying these gaps enables data leaders to prioritize initiatives and create a realistic roadmap for implementing the data strategy.</p> </section> <section class="section main-article-chapter" data-menu-title="3. Specify the desired state for data management and analytics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Specify the desired state for data management and analytics</h2> <p>Once the current-state assessment is complete and the results have been evaluated, articulate what works well and where changes are needed in data management and analytics processes. Defining the desired state clarifies what the organization can achieve through those changes. This vision should be based on the previously identified business imperatives for each data domain and function.</p> <p>As part of this step, set data quality expectations and outline plans to harmonize core data management processes, such as data integration, metadata management and master data management. Doing so ensures consistency across systems and reliable access to <a href="https://www.techtarget.com/searchdatamanagement/opinion/Trusted-data-is-the-foundation-of-data-driven-decisions-GenAI">relevant and trustworthy data</a>.</p> </section> <section class="section main-article-chapter" data-menu-title="4. Identify and prioritize critical data domains"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Identify and prioritize critical data domains</h2> <p>The strategic value of data varies. While an enterprise data strategy by definition should address all data domains, focus initial implementation efforts on the domains and associated data sets that are most critical to business operations and decision-making.</p> <p>Identifying and prioritizing the highest‑value data domains enables data leaders to direct resources to areas where data management and analytics improvements will have the greatest business impact. In a retailer, for example, improving customer data quality enables more accurate analytics for targeted marketing and better customer service. Focusing on high‑value areas also helps demonstrate the data strategy's value and build momentum toward a more <a href="https://www.techtarget.com/searchdatamanagement/tip/Use-these-steps-to-successfully-build-your-data-culture">data‑centric culture</a>.</p> </section> <section class="section main-article-chapter" data-menu-title="5. Create an implementation roadmap"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. Create an implementation roadmap</h2> <p>After defining what your organization aims to achieve with data to support business priorities and what's required to do so, create an implementation roadmap that details how it will get there. A well-designed roadmap sequences data initiatives over time in a way that's achievable and measurable.</p> <p>That requires balancing ambition with realism to enable sustained, disciplined progress on the data strategy rather than a series of disconnected short-term projects -- or, worse, overpromising on planned deployments. The roadmap should also connect long-term goals, such as becoming more data-driven or AI-enabled, to concrete steps across data management and analytics processes.</p> </section> <section class="section main-article-chapter" data-menu-title="6. Develop data principles and strategic guardrails"> <h2 class="section-title"><i class="icon" data-icon="1"></i>6. Develop data principles and strategic guardrails</h2> <p>Incorporate data principles and strategic guardrails into the data strategy so they actively shape decisions on data management and use, rather than being abstract guidelines. Foundational principles -- such as treating data as an enterprise asset, ensuring it's accurate and accessible, and establishing a single source of truth through transparent data management practices -- should directly inform the data operating model and architecture. This drives data consistency, reuse and trust across the organization.</p> <p>Strategic guardrails are operational constraints and requirements in areas such as privacy, security, <a href="https://www.techtarget.com/searchbusinessanalytics/feature/Why-ethical-use-of-data-is-so-important-to-enterprises">ethical data use</a>, data quality and data platform design. Embed them in the data strategy as part of data governance policies and the implementation roadmap. Aligning suitable guardrails with the execution of data initiatives provides clear direction on appropriate data use, reduces data-related risks and enables BI, data science and business teams to innovate confidently within well-defined boundaries.</p> </section> <section class="section main-article-chapter" data-menu-title="7. Build a data governance framework and assign data ownership"> <h2 class="section-title"><i class="icon" data-icon="1"></i>7. Build a data governance framework and assign data ownership</h2> <p>A strong data governance program is a <a href="https://www.techtarget.com/searchdatamanagement/tip/6-key-components-of-a-successful-data-strategy">critical component of a data strategy</a>. Effective data governance ensures that data remains consistent and reliable and that it's managed and used properly. Without it, various problems can arise. For example, different departments might create conflicting data definitions or data quality might deteriorate, compromising business decisions due to incomplete or inaccurate information.</p> <p>Include <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">implementing the data governance framework</a> as a foundational item in the data strategy's roadmap. The strategy should also detail expectations for managing data throughout its lifecycle and the role of data governance in supporting business objectives. Additionally, work with business stakeholders to assign ownership of data assets to appropriate individuals or teams and task them with ensuring the data they oversee is managed and used in accordance with governance policies.</p> </section> <section class="section main-article-chapter" data-menu-title="8. Design an enterprise data architecture"> <h2 class="section-title"><i class="icon" data-icon="1"></i>8. Design an enterprise data architecture</h2> <p>A <a href="https://www.techtarget.com/searchdatamanagement/definition/What-is-data-architecture-A-data-management-blueprint">data architecture</a> provides the technical foundation for managing and delivering data. It defines and visualizes how data is processed, integrated, stored and accessed across systems. However, in many organizations, the existing data architecture has been developed over time, often in a piecemeal fashion without an enterprise-wide focus. As a result, redundancies and gaps in the architecture create challenges with data access and use.</p> <p>To address these issues, design an enterprise data architecture as part of the data strategy. In addition to a high-level architectural blueprint, it should include artifacts such as data models, data flow diagrams and documents that map data use to business processes. A well-designed data architecture guides data management processes, helps teams identify data challenges and supports both operational reporting and advanced analytics.</p> </section> <section class="section main-article-chapter" data-menu-title="9. Implement security, privacy and regulatory compliance controls"> <h2 class="section-title"><i class="icon" data-icon="1"></i>9. Implement security, privacy and regulatory compliance controls</h2> <p>Protecting the ever-increasing volumes of data that organizations collect and use is critical to avoiding business problems. In addition to strategic guardrails that set high-level boundaries on data management and use, a data strategy must include specific controls to <a href="https://www.techtarget.com/searchdatamanagement/feature/Top-3-data-privacy-challenges-and-how-to-address-them">mitigate data security and privacy risks</a>. For example, ensure that only authorized users can access sensitive data and that potential security threats can be detected and addressed quickly through predefined incident response plans.</p> <p>Regulatory compliance is also a broader issue now due to the <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know">growing number of data protection laws</a> that require responsible management of personal information and transparency about how data is used. Integrate compliance mechanisms into the data strategy to help reduce legal risks and maintain trust with customers and business partners.</p> </section> <section class="section main-article-chapter" data-menu-title="10. Enable data accessibility and increased data literacy"> <h2 class="section-title"><i class="icon" data-icon="1"></i>10. Enable data accessibility and increased data literacy</h2> <p>Making trusted data accessible to the people who need it is a core objective of an enterprise data strategy. Data access is no longer restricted to technical specialists. A modern data strategy supports controlled, governed access for business users and data analysts through user-friendly dashboards, self-service analytics tools and <a href="https://www.techtarget.com/searchdatamanagement/answer/What-steps-are-key-to-building-a-data-catalog">centralized data catalogs</a>.</p> <p>However, data accessibility alone isn't enough. Increased data literacy is also required across the organization to maximize the business value derived from data assets. As part of the data strategy, <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Develop-a-data-literacy-program-to-fit-your-company-needs">develop a data literacy program</a> that sets expectations for workers and includes training to help them become more data-literate.</p> </section> <section class="section main-article-chapter" data-menu-title="11. Build in support for BI, advanced analytics and AI applications"> <h2 class="section-title"><i class="icon" data-icon="1"></i>11. Build in support for BI, advanced analytics and AI applications</h2> <p>In the past, data strategies often focused primarily on delivering data for use in BI and reporting applications. But now they must also focus on the data needed for expanding deployments of advanced analytics and AI applications in companies.</p> <p>Build support for techniques and tools such as predictive analytics, machine learning and both generative AI and agentic AI into the enterprise data strategy. Used effectively, they help organizations identify patterns in large data sets, forecast trends, explore data more efficiently and optimize or automate business processes. However, they <a href="https://www.techtarget.com/searchdatamanagement/opinion/The-future-of-AI-depends-on-better-data-not-bigger-models">depend on the strong data foundation</a> that a data strategy provides.</p> </section> <section class="section main-article-chapter" data-menu-title="12. Define metrics to track and evolve the data strategy"> <h2 class="section-title"><i class="icon" data-icon="1"></i>12. Define metrics to track and evolve the data strategy</h2> <p>A data strategy should evolve over time as business priorities, data sets, technologies and regulations change -- and as problems are identified. To guide this evolution, define <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-metrics-Data-quality-data-literacy-and-more">KPIs and other metrics</a> to track the effectiveness of data initiatives. Include ones on things such as data quality, governance activities and data availability, security and use.</p> <p>Monitoring them enables data and business leaders to evaluate progress on initiatives and identify areas for improvement. Regular reviews and continuous refinement of the data strategy ensure that it remains aligned with the organization's needs and continues to deliver business value. Spell out the need for that upfront, when setting expectations for the strategy, so it isn't a surprise to anyone.</p> <p><em>Anne Marie Smith, Ph.D., is an information management professional and consultant with broad experience across industries. She has also designed and delivered numerous data management courses and educational programs.</em></p> </section> Here are 12 to-do items for data leaders developing a data strategy to help their organization use data more effectively for analytics and business decision-making. https://cdn.ttgtmedia.com/visuals/search400/iseries_database_manage/search400_article_014.jpg https://www.techtarget.com/searchdatamanagement/tip/Developing-an-enterprise-data-strategy-10-steps-to-take Fri, 17 Apr 2026 15:50:00 GMT <p>Every business today is a data business. From the corner store tracking stock levels to the multinational manufacturer predicting market trends and shipping costs worldwide, all businesses run on data.</p> <p>Specifically, they run on many types of data. For example, businesses of all kinds have transaction, reference and customer relationship data. We might also have industry-specific and external data, as well as metadata describing their formats and uses. Often, we integrate all these data types to create specialized analytics data sets. A well-planned data strategy keeps this complex ecosystem in order, with a <a href="https://searchdatamanagement.techtarget.com/tip/5-principles-of-a-well-designed-data-architecture">strong data architecture</a> as its foundation.</p> <section class="section main-article-chapter" data-menu-title="Why do you need a data strategy?"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why do you need a data strategy?</h2> <p>A data strategy defines long-term objectives for how an organization uses data, along with the policies and practices that support them. To be successful, a data strategy must cover all data use cases – not just technical processes for <a href="https://searchdatamanagement.techtarget.com/definition/data-management">data management</a> and analytics, but also the human element.</p> <p>No modern business can leave the management, security and use of such an important corporate asset to individual data architects or developers. A comprehensive data strategy, with broad involvement and support, ensures data is managed well and used effectively.</p> <p>Data priorities differ across organizations, shaped by management strategies and business goals, so there's no generic template to follow. But there are six critical components every data strategy must include.</p> </section> <section class="section main-article-chapter" data-menu-title="1. Data"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Data</h2> <p>This is the most fundamental component, of course. But all the advice that follows will be of no help if your data isn't safely stored and secured, well-maintained and ready for use. The strategic value of your data must be built on a solid base of enterprise data management. That includes integrating and processing your data, <a href="https://www.techtarget.com/searchdatamanagement/tip/6-dimensions-of-data-quality-boost-data-performance">validating its quality</a>, governing its use and auditing the processes that affect it.</p> <p>Once these basics are in place, I always recommend an <a href="https://searchdatamanagement.techtarget.com/answer/What-steps-are-key-to-building-a-data-catalog">enterprise data catalog</a> as a critical component of a data strategy. You can't strategize around data if you don't know what data you have. Data catalog tools are particularly useful for making data available to business users by providing detailed, descriptive metadata. Sometimes IT managers want to map their systems -- to know what data they have and where it resides. The IT team can create its own simplified data catalog for such needs.</p> <p>The key questions are always the same. What data do I have? Where is it? Who can use it?</p> </section> <section class="section main-article-chapter" data-menu-title="2. Tools"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Tools</h2> <p>Data catalog tools are provisioned by IT and data management teams who know how to use the various features in <a href="https://www.techtarget.com/searchdatamanagement/feature/16-top-data-catalog-software-tools-to-consider-using">data catalog software</a>, set them up and deploy them. We can make a useful distinction between tools provided by IT and tools adopted by end users. Both play an important role in a data strategy, complementing rather than contradicting each other.</p> <p>Data management tools are almost always the domain of IT. There are some lightweight data quality and data integration tools designed for business users, but data management remains largely a behind-the-scenes function.</p> <p>IT often also deploys the BI tools used to create data visualizations, dashboards and reports. But data and business analysts might have their own preferences and choose different tools. That can work well so long as we put controls in place to <a href="https://searchbusinessanalytics.techtarget.com/feature/Data-governance-framework-key-to-analytics-success">govern data access and usage</a>. Likewise, data scientists might feel most comfortable using tools they already mastered or that support certain analytics methodologies.</p> <p>In the past, most IT teams tried to prevent the use of unsanctioned, non-standard tools. Now, just as we've adapted to bring-your-own-device, analytics specialists commonly bring their own favored applications. A good data strategy embraces that diversity but with sensible limits. In this case, we can ask another question: What tools are appropriate to use? Enabling a data analyst to use a <a href="https://searchbusinessanalytics.techtarget.com/definition/self-service-business-intelligence-BI">self-service BI</a> application to build some dashboards is reasonable; allowing someone to build their own data warehouse beyond their skills and authority is not.</p> <figure class="main-article-image full-col" data-img-fullsize="https://www.techtarget.com/rms/onlineImages/data_management-key_stages_data_strategy-f.png"> <img data-src="https://www.techtarget.com/rms/onlineImages/data_management-key_stages_data_strategy-f_mobile.png" class="lazy" data-srcset="https://www.techtarget.com/rms/onlineImages/data_management-key_stages_data_strategy-f_mobile.png 960w,https://www.techtarget.com/rms/onlineImages/data_management-key_stages_data_strategy-f.png 1280w" alt="Key stages of the data strategy development process" height="267" width="560"> <figcaption> <i class="icon pictures" data-icon="z"></i>These are the four main phases of developing a data strategy, according to Donna Burbank of Global Data Strategy. </figcaption> <div class="main-article-image-enlarge"> <i class="icon" data-icon="w"></i> </div> </figure> </section> <section class="section main-article-chapter" data-menu-title="3. Analytics techniques"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Analytics techniques</h2> <p>Just as we use various analytics tools depending on our needs, we also employ a variety of analytics techniques. Data visualization is a common example. We might also find uses for predictive analytics, text analytics, sentiment analysis and cluster analysis, to name a few advanced analytics techniques. They can be powerful and useful, but also need careful oversight. Without it, we might run afoul of <a href="https://searchdatamanagement.techtarget.com/definition/data-governance">data governance</a> and privacy laws.</p> <p>Predictive analytics, for example, might show business value in optimizing equipment maintenance cycles. That's an uncontroversial use. But predictive techniques could also be used to help automate hiring or manage marketing promotions. In those cases, employees and consumers might have concerns about the reliability, fairness or openness of the process.</p> <p>A data strategy must recognize that governing only data and tools might not suffice. We need to understand -- and train people to understand -- that not all analytics techniques are neutral. Some use cases, especially those involving personally identifiable information, won't be justified by their business value alone.</p> </section> <section class="section main-article-chapter" data-menu-title="4. Collaboration"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Collaboration</h2> <p>In modern businesses, data use is typically more collaborative than in the past. Increased data literacy and easier-to-use tools mean more people can participate in analytics, as well as technical fields like <a href="https://searchbusinessanalytics.techtarget.com/definition/data-preparation">data preparation</a> and data quality.</p> <p>Even closely controlled processes, such as data governance and primary data definition development, can be crowdsourced. For example, doing so can ensure that product names, error codes and managed processes reflect reality on the shop floor in a manufacturing company. Collaboration on primary data can also avoid that most frustrating customer service response: "There's no code for that."</p> <p>Collaborative tools are also being used more, including file sharing, enterprise chat, messaging and video conferencing. Human beings are compulsive collaborators. We constantly share, discuss and debate with others. If collaboration isn't planned for, it will happen anyway -- unplanned.</p> <p>Consider the role of data and analytics in your organization's business decisions and identify processes that involve engagement within and beyond teams. Use that insight to support the ability to share and comment on dashboards, reports and data visualizations.</p> <p>For example, some <a href="https://searchbusinessanalytics.techtarget.com/feature/How-to-evaluate-and-select-the-right-BI-analytics-tool">BI and analytics tools</a> enable multiple users to annotate visualizations. Increasingly, they also integrate with chat and messaging apps. Even simple file sharing can be effective, especially when supported by enterprise-class scalability and security features.</p> </section> <section class="section main-article-chapter" data-menu-title="5. Documentation and auditing"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. Documentation and auditing</h2> <p>In describing these data strategy components, I've emphasized the need to balance IT control with end users' freedom to do self-service when appropriate.</p> <p>To find this balance, our strategic goals must be well documented. Successful data strategies are built on the ability to answer four questions about any element of the plan and any resource -- data, tools, etc. -- it incorporates.</p> <ul class="default-list"> <li>What is appropriate?</li> <li>What is approved?</li> <li>What is the purpose?</li> <li>What is the governance policy?</li> </ul> <p>With good documentation of both the data strategy and the underlying data architecture, we can answer these questions before any new project or initiative. We should also be able to look back at any project and answer them retrospectively. By doing so, we put ourselves in a good position to audit how the data strategy is working. It can also help us assess compliance with data governance policies and other <a href="https://searchdatamanagement.techtarget.com/feature/Data-model-design-tips-to-help-standardize-business-data">internal data standards</a>.</p> <div class="youtube-wrapper"> <iframe width="560" height="315" src="https://www.youtube.com/embed/BqdPuwvwPk4?rel=0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> </div> </section> <section class="section main-article-chapter" data-menu-title="6. People"> <h2 class="section-title"><i class="icon" data-icon="1"></i>6. People</h2> <p>The two most important elements of your data strategy are the bookends of this list: data and people. Organizations increasingly look for <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Data-literacy-training-requires-a-dual-approach">data literacy and analytics skills</a> in new business hires. Almost every business school now teaches basic data analytics.</p> <p>Data scientists remain in high demand, though the role has evolved significantly in recent years. AI and machine learning have reshaped what organizations require. The priority now is professionals who can not only analyze data, but also build and govern the systems that act on it.</p> <p>You should also think carefully about IT and data management in your staffing and hiring. With so much technology running in the cloud and systems more robust than ever, it's tempting to think IT merely has to keep the lights on. It's not true. High availability, disaster recovery, meeting service-level agreements, supporting new business requirements and regulatory demands all fall into IT's domain.</p> <p>Data architects, data integration developers, data engineers, database administrators and other <a href="https://searchdatamanagement.techtarget.com/feature/Data-management-roles-Data-architect-vs-data-engineer-others">data management professionals</a> also play key roles in meeting business needs. An IT staff that is savvy about the business is a great strategic advantage. That caliber of IT staff needs recognition and leadership support as much as any other role.</p> </section> <section class="section main-article-chapter" data-menu-title="How to implement an effective data strategy"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How to implement an effective data strategy</h2> <p>These six key components aren't a complete guide to <a href="https://www.techtarget.com/searchdatamanagement/tip/Developing-an-enterprise-data-strategy-10-steps-to-take">developing a data strategy</a>. You also must consider broader concerns, such as budgets, competition, innovation, marketing plans, staffing policies and legal frameworks.</p> <p>But you can <a href="https://theodi.org/article/data-strategy-how-an-ecosystem-approach-can-help-shape-your-vision/">apply this thinking broadly</a>. For example, your staffing plan could include guidelines for making better use of data and analytics based on strategic priorities. Product innovation is increasingly driven by data on customer feedback, user behavior and market trends.</p> <p>Implementing a data strategy requires understanding your entire organization's strategic goals. From there, break down the role of data, how it will be managed and used and apply it consistently across production, finance, marketing and HR. The result will be a data strategy that is workable and flexible for ever-changing business pressures and needs.</p> <p><strong>Editor's note:</strong>&nbsp;<em>This article was republished in April 2026 to improve the reader experience.&nbsp;</em></p> <p><em>Donald Farmer is a data strategist with 30+ years of experience, including as a product team leader at Microsoft and Qlik. He advises global clients on data, analytics, AI and innovation strategy, with expertise spanning from tech giants to startups. He lives in an experimental woodland home near Seattle.</em></p> </section> These six elements are essential parts of an enterprise data strategy that will help meet business needs for information when paired with a solid data architecture. https://cdn.ttgtmedia.com/rms/onlineimages/strategy_a200792738.jpg https://www.techtarget.com/searchdatamanagement/tip/6-key-components-of-a-successful-data-strategy Wed, 15 Apr 2026 09:00:00 GMT <p>AI made semantic search mainstream. Now, enterprise reality is forcing a strategic refinement.</p> <p>Vector search became a common requirement after the release of ChatGPT and the rise of generative AI chatbots, and it's now a standard feature across many database platforms. But increasingly, vector search is no longer the sole decision point for data leaders looking for effective search and retrieval frameworks to support AI applications. As usage expands into business‑critical workflows, some implementations can strain under relevance gaps and rising operational and governance overhead. What matters now is hybrid search, which combines semantic similarity with keyword precision, because enterprise queries often require both meaning and exact terms. That shift is pushing many organizations to update their search approaches to match real business use.</p> <section class="section main-article-chapter" data-menu-title="Why early vector search deployments break down"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why early vector search deployments break down</h2> <p>The practical problem is that many first-generation deployments struggle <a href="https://www.techtarget.com/searchdatamanagement/tip/Assemble-the-layers-of-big-data-stack-architecture">as AI initiatives expand and data volumes grow</a>. A system might handle similarity search reasonably well but stumble on terms specific to the business, such as product names, acronyms, customer identifiers, error codes and policy language. As those misses add up, it's time to reassess the type of search and retrieval architecture needed to fully support a more mature environment.</p> <p>Hybrid search, also referred to as hybrid retrieval, sits at the center of these discussions because it reflects how enterprise search for AI applications works. Some queries depend on exact matches, <a href="https://www.techtarget.com/searchdatamanagement/opinion/Why-data-semantics-matters-for-context-aware-systems">others on semantic similarity</a>, and many require both. Hybrid search runs full-text and vector queries in parallel and blends the results into a single ranked list.</p> <p>For database buyers, it's clear that hybrid search is the baseline. Standalone vector database products still have their place, and many also now support full-text search. But many teams can <a href="https://www.techtarget.com/searchdatamanagement/feature/AI-data-governance-guidance-that-gets-you-to-the-finish-line">store and query vector embeddings</a> in their current systems, including core database engines and managed services. More and more, platform differentiation comes down to relevance, including filters that narrow results to the right scope and reranking capabilities that push the best candidates to the top of search results.</p> </section> <section class="section main-article-chapter" data-menu-title="Three ways to modernize a retrieval framework"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Three ways to modernize a retrieval framework</h2> <p>When a framework change is needed to help the business gain the expected benefits from AI applications, organizations have three options.</p> <h3>1. Extend the existing platform</h3> <p>The first path is to extend an existing data platform. This is usually the right move when vector-based retrieval is primarily an upgrade to the infrastructure the organization already uses and trusts.</p> <p>The goal is to keep retrieval within the existing data stack while improving search support for AI workloads. MongoDB Atlas, Databricks, Snowflake, Azure Cosmos DB and PostgreSQL with the pgvector extension fit this pattern because vector search is <a href="https://www.techtarget.com/searchdatamanagement/feature/Evaluating-the-different-types-of-DBMS-products">integrated into the broader platform</a>, rather than deployed as a separate system.</p> <p>For buyers, this path tends to make the most sense when governance continuity, platform simplicity and reusing the skills of existing operational teams matter more than introducing another specialized layer.</p> <h3>2. Upgrade the search layer</h3> <p>The second path is to upgrade the search layer. If most complaints focus on the search experience, the decision is less about the database and more about adopting a search-first layer optimized for relevance at scale.</p> <p>Search-first platforms are typically designed around the idea that retrieval quality comes from combining full-text search and <a href="https://www.techtarget.com/searchenterpriseai/tip/Top-RAG-tools">vector-based similarity search</a> with ranking and filtering across indexed content. Azure AI Search, Elasticsearch, OpenSearch, Apache Solr and Algolia belong in this broader category.</p> <p>This path is best when the enterprise needs stronger discovery, ranking and search quality across data sets, documents, knowledge bases, websites and other types of content.&nbsp;</p> <h3>3. Replace the existing vector platform or add a dedicated one</h3> <p>The third path is to replace or add a specialized vector platform. This should usually be the escalation path, not the default.</p> <p>While specialized platforms such as Pinecone, Weaviate, Qdrant and&nbsp;Milvus (including the Zilliz Cloud service) also offer hybrid search capabilities, they are centered on dedicated vector retrieval infrastructure. That can make sense when retrieval has become strategic enough to justify a separate platform, or when current data and search environments no longer fit the workload.</p> <p>A few use cases help clarify the three options. An enterprise building an internal knowledge assistant might not need a new platform if it can extend its existing data stack and improve search quality there. A company with a large digital content estate and weak search relevance might get more value from modernizing the search layer than from reworking the entire data platform. And a business that uses retrieval as a service across multiple AI products might decide it needs a dedicated vector platform. &nbsp;</p> </section> <section class="section main-article-chapter" data-menu-title="Why relevance and governance steer retrieval choices"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why relevance and governance steer retrieval choices</h2> <p>Search and retrieval for AI are no longer just features that organizations set and forget. It is a core capability, and buying decisions now extend beyond whether a platform supports vector search.</p> <p>For many organizations, the primary requirement is relevance quality. Can the platform support intent‑driven search while still returning precise results for specific business terms? Hybrid search has become the baseline by combining semantic understanding and keyword matching in a single request.</p> <p>The second buying criterion is governance fit. Search and retrieval more frequently touch regulated, sensitive and business‑critical data. Can the platform work within the organization's governance model, rather than forcing new controls or workarounds?</p> <p>The governance requirement is <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-for-AI-requires-a-cross-functional-approach">only getting sharper as AI expands</a>. A March 2026 Omdia <a target="_blank" href="https://research.esg-global.com/reportaction/515202164/Toc" rel="noopener">report</a> said 47% of 400 technical and business stakeholders cited data privacy as their organization's top risk in a generative AI initiative, and 38% underestimated security and governance costs. (Omdia is a division of Informa TechTarget.) Gartner's November 2025 report on cloud database management systems complements Omdia's findings, noting that metadata is emerging as the connective tissue for AI and search workflows.</p> <p>As platforms move toward data fabrics and self-governing systems, integrated metadata becomes central to governance, observability and operational control, requiring a search and retrieval platform that improves over time and holds up under evolving AI workloads.</p> <p><i>Tom Walat is an editor and reporter for TechTarget, where he covers data technologies.</i></p> </section> As AI workloads mature, enterprises face multiple data platform choices to improve search and retrieval capabilities while meeting governance and operational demands. https://cdn.ttgtmedia.com/rms/onlineimages/code_g684641103.jpg https://www.techtarget.com/searchdatamanagement/feature/Hybrid-search-demands-reshape-retrieval-frameworks-for-AI Tue, 14 Apr 2026 18:04:00 GMT <div> <div></div> </div> <p>As regulatory landscapes evolve in an increasingly data-driven world, organizations face increasing pressure to ensure compliance.</p> <p>Data-specific requirements govern <a href="https://www.techtarget.com/searchdatamanagement/feature/Big-data-collection-processes-challenges-and-best-practices">how organizations collect, store, process and share data</a>. Achieving compliance is an essential, ongoing activity that leadership must guide. To do so, executives must understand the various global regulatory requirements and the implications and risks of non-compliance. Successful compliance shows that the enterprise and its personnel fully embrace data governance across all data-related activities.</p> <section class="section main-article-chapter" data-menu-title="Important data governance regulations"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Important data governance regulations</h2> <p><a href="https://www.techtarget.com/searchdatamanagement/definition/data-governance">Data governance</a> is an umbrella term encompassing several important activities, including data lifecycle, stewardship, security, privacy, destruction, quality, retention, access, classification and management. Identifying the specific enterprise data governance requirements under each regulation is essential for leadership, especially as their business expands internationally.</p> <p>The following are data governance regulations that affect <a href="https://www.techtarget.com/searchdatamanagement/feature/How-agentic-AI-governance-tackles-data-security-challenges">governance planning</a>, strategies and procedures. Most of these laws apply to any organization doing business in the country, regardless of origin. &nbsp;</p> <ul class="default-list"> <li><b>EU GDPR – </b>A pivotal piece of data protection legislation, <a href="https://www.techtarget.com/whatis/definition/General-Data-Protection-Regulation-GDPR">GDPR</a> protects EU residents' personal data. It specifies data management strategies that organizations must follow, including conducting a data protection impact assessment to identify and address any risks. Failure to comply may result in significant financial penalties -- up to €20 million ($23 million) or 4% of the firm's worldwide annual revenue.</li> <li><b>CCPA – </b>Consumers have the right to know how organizations collect and process their data. <a href="https://www.techtarget.com/searchcio/definition/California-Consumer-Privacy-Act-CCPA">CCPA</a> ensures California residents have the right to delete or limit the personal information organizations collect, to opt out of the sale of their data and correct inaccurate information. Violators can range from $2,663 (unintentional) to $7,988 (intentional).</li> <li><b>UK GDPR and Data Protection Act – </b>Enacted in 2018, <a href="https://www.techtarget.com/searchdatabackup/definition/Data-Protection-Act-2018-DPA-2018">this legislation</a> transposes the GDPR into UK law.<b> </b>It requires strong data security, collection and processing practices. Penalties can range from £8.7 million ($11.5 million) to £17.5 million ($23.1 million), or 2% to 4% of the company's worldwide annual revenue -- whichever is higher.</li> <li><b>HIPAA – </b><a href="https://www.techtarget.com/searchhealthit/definition/HIPAA">HIPAA Security and Privacy Rules</a> apply specifically to the US healthcare system and govern rules on data access, security, use, and protected health information disclosures. It requires risk assessments and employee training. Violations are either civil or criminal, and penalties vary based on severity. Unknowing civil offenders face fines as low as $100 per violation, while willful offenders face fines up to $50,000 per violation. Criminal incidents can result in a fine of up to $250,000 and 10 years in prison.</li> <li><b>EU Data Governance Act – </b>Launched in 2023, this legislation requires secure data sharing across the EU. It advocates data altruism, which examines how data can be used in the public interest. The act doesn't specify a blanket fine but offers criteria for determining penalties.</li> <li><b>Sarbanes-Oxley Act (SOX) – </b><a href="https://www.techtarget.com/searchcio/definition/Sarbanes-Oxley-Act">SOX legislation</a> addresses issues in<b> </b>financial management and reporting as applicable to all publicly traded companies in the US. It has strict controls on the accuracy, integrity, validation and verification of financial data. It also <a href="https://www.techtarget.com/searchcio/definition/What-is-SOX-compliance-A-complete-guide-and-checklist">mandates effectiveness assessments</a> for internal controls and data governance practices. Violators face 10 to 20 years in prison and hefty fines.</li> <li><b>UK Network and Information Systems regulations – </b>These regulations focus on cybersecurity and incident reporting for network and information services providers. Cybersecurity requirements include regular security assessments and continuous improvements. Penalties cost up to £17 million ($22.4 million).</li> <li><b>Gramm-Leach-Bliley Act (GLBA) – </b><a href="https://www.techtarget.com/searchcio/definition/Gramm-Leach-Bliley-Act">This US legislation</a> mandates financial organizations establish information disclosure policies, implement security programs and perform regular risk assessments. Noncompliance can result in a $100,000 fine per violation.</li> <li><b>Personal Information Protection Law (PIPL) –</b> China's data protection law is among the toughest globally, applying to all enterprises handling personal data within China's borders. It has strict consent and trans-border data flow requirements. Penalties for non-compliance include ¥50 million RBM ($7 million), 5% of annual revenue or shutting down enterprises.</li> <li><b>Digital Personal Data Protection Act (DPDPA) –</b> India's 2023 act requires data fiduciaries to provide customers notices of their rights and inform them of the type of data they're collecting and why, with specific restrictions on cross-border data flows. The <a href="https://www.techtarget.com/searchdatabackup/definition/Digital-Personal-Data-Protection-Act-2023">DPDPA</a> mandates consent for any processing, with additional requirements regarding children's data. Penalties include up to ₹250 crore ($26.9 million).</li> <li><b>Personal Data Protection Act –</b> Developed in Singapore, thi<i>s </i>legislation is widely recognized throughout the Asia-Pacific region. It is consent-driven, mandates breach alerts and has retention limitations. If a company exceeds S$10 million ($7.7 million) in annual turnover in Singapore, it faces financial penalties up to 10% of that annual turnover. Otherwise, fines cannot exceed S$1 million ($778,000).</li> <li><b>Personal Data Protection Law –</b> The UAE law regulates personal data processing, requiring consent and security, as well as strict rules for trans-border data flows. It gives individuals the right to correct inaccuracies and stop processing upon request. Noncompliance results in fines up to AED 5 million ($1.36 million).</li> <li><b>Law 0908 on Personal Data Protection – </b>Morocco's legislation is one of Africa's most comprehensive data protection statutes. It requires organizations to register with the national government. Penalties for noncompliance include fines up to MAD 600,000 ($64,343) and/or imprisonment from three months to four years.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Non-compliance risks for executives"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Non-compliance risks for executives</h2> <p paraeid="{2c1fe5c8-d26a-421b-895d-711c4f19c464}{16}" paraid="2110754361">While data governance is very much a technology-centered activity, it is also an executive accountability issue. If <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">data governance initiatives</a> result in regulatory violations, improper AI use or data-related incidents, the highest levels of enterprise leadership -- including the C-suite and the board -- are liable. Penalties include fines, litigation, reputational damage and competitive risks.</p> </section> <section class="section main-article-chapter" data-menu-title="Compliance resources"> <h2 class="section-title"><i class="icon" data-icon="1"></i><iframe title="Risks of non-compliance with data regulations" aria-label="Table" id="datawrapper-chart-wuECR" src="https://datawrapper.dwcdn.net/wuECR/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="819" data-external="1"></iframe><span xml:lang="EN-US" data-contrast="none"><span data-ccp-parastyle="heading 2"></span></span><em></em>Compliance resources</h2> <p>Many organizations have created <a href="https://www.techtarget.com/searchdatamanagement/tip/5-data-governance-framework-examples">data governance frameworks</a> that help enterprises establish data governance capabilities, including the following:</p> <ul class="default-list"> <li><b>Data Management Body of Knowledge (DAMA-DMBOK) – </b>Considered the industry standard for data governance, DAMA-DMBOK addresses data quality, stewardship and metadata, among other issues.</li> <li><b>Control Objectives for Information and Related Technologies (COBIT) – </b>Developed by ISACA, <a href="https://www.techtarget.com/searchsecurity/definition/COBIT">COBIT</a> offers strong controls and audit guidelines that align IT governance with business risk management and strategy.</li> <li><b>NIST Cybersecurity &amp; Privacy Frameworks – </b>NIST has two data governance frameworks: the<b> </b><a href="https://www.techtarget.com/searchsecurity/definition/NIST-Cybersecurity-Framework">Cybersecurity Framework</a> for reducing cybersecurity risks<b> </b>and<b> </b>the Privacy Framework to identify and manage privacy risks.</li> <li><b>ISO/IEC 38500 – </b>Most recently updated in 2024, this standard is a key international standard for IT governance. It addresses legal, regulatory and ethical data use and provides vocabulary for IT governance.</li> <li><b>Data Management Capability Assessment Model (DCAM) – </b>Developed by the EDM Council, this framework defines a maturity model addressing data governance, quality and architecture.</li> </ul> <p>A <a href="https://www.techtarget.com/searchdatamanagement/feature/15-top-data-governance-tools-to-know-about">variety of tools and resources</a> can help demonstrate compliance, including master data management tools, data discovery and classification tools, data catalogs and IAM systems. Senior management support and budget funding are essential for establishing a mature data governance program.</p> <p>Consider investing in AI tools, which can greatly improve performance, provide better data analytics, automate repetitive processes and identify potential compliance issues. Existing tools and resources might have upgraded versions with AI capabilities.</p> </section> <section class="section main-article-chapter" data-menu-title="How to achieve compliance"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How to achieve compliance</h2> <p>The following are best practices for executives to achieve optimal data governance compliance outcomes.</p> <h3>Be accountable for and own data governance</h3> <p>Just as organizations should have data owners and stewards for different domains, they should also <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-responsibilities-now-belong-in-the-C-suite">make an executive responsible for data governance</a> and compliance activities. Responsibilities include defining and measuring KPIs, conducting periodic board-level governance briefings and establishing partnerships with other departments, such as legal, HR, risk management and operations.</p> <h3>Ensure that data governance is risk-based</h3> <p>Establish data governance as a primary risk area. Add governance to a corporate risk register and examine risk from financial and regulatory perspectives. Map governance controls to appropriate regulations and frameworks. Building scenarios to address specific risk events, such as trans-border data violations, will help if they ever occur.</p> <h3>Require auditable evidence on compliance activities</h3> <p>Demonstrating data governance compliance at any time is essential in case of unannounced audits. Evidence of compliance includes project reports, compliance testing results, access management issues, <a href="https://www.techtarget.com/searchdatamanagement/tip/Evaluating-data-quality-requires-clear-and-measurable-KPIs">data quality measurements</a> and retention/deletion rules. Schedule quarterly audits for relevant controls and create evidence trails for regulator inquiries.</p> <h3>Optimize data quality at the C-Level</h3> <p>Data quality and lineage must be primary goals. Establish strong controls addressing <a href="https://www.techtarget.com/searchdatamanagement/feature/How-data-lineage-became-a-boardroom-metric">data quality, lineage and accuracy</a>. Enforce data quality standards, launch quality checks and link metrics to business requirements. Establish beginning-to-end data lineage and ensure access to it.</p> <h3>Enforce data access controls</h3> <p>Senior leaders must ensure data access controls are consistently monitored, enforced and applied. Implement least privilege, role-based access controls, multi-factor authentication, segregation of duties and uninterrupted monitoring. Provide support for potential audits.</p> <h3>Culture of compliance</h3> <p>This starts with the C-suite and board. Mandate training for all employees on data-related activities and endorse data literacy throughout the enterprise. Regularly reiterate the importance of data governance at major company meetings. Support whistleblowing of any violations and note governance issues in performance reviews.</p> <p>Linking all governance activities into a cohesive process also helps with compliance.<b> </b>Data silos can spell disaster. Greater information sharing, along with the integration of security and privacy capabilities across systems, helps avoid this.</p> <h3>Acquire technology that facilitates the compliance process</h3> <p>The right technology ensures that governance activities are scalable, adaptable and automated. Automate data governance activities by integrating <a href="https://www.techtarget.com/searchdatamanagement/feature/AI-data-governance-guidance-that-gets-you-to-the-finish-line">AI tools</a> with risk, privacy and security systems. However, be sure to provide <a href="https://www.techtarget.com/searchenterpriseai/definition/AI-governance">AI governance</a> oversight. When used correctly, it can facilitate the following data processes:</p> <ul class="default-list"> <li>Reduce the likelihood of human error.</li> <li>Improve performance.</li> <li>Automate repetitive tasks such as data collection and classification.</li> <li>Identify potential compliance issues.</li> <li>Deliver reports for auditors.</li> <li>Ensure cross-border flow adheres to regulations.</li> </ul> <h3>Ensure that knowledge of regulatory activities is current</h3> <p>Adapting to <a href="https://www.techtarget.com/searchsecurity/tip/State-of-data-privacy-laws">regulatory changes</a> and maintaining compliance are essential for enterprises. Executives should consistently monitor the global regulatory landscape. Review and assess regulatory changes, keep policies current and train governance teams to do the same.</p> <p><em>Paul Kirvan, FBCI, CISA, is an independent consultant and technical writer with more than 35 years of experience in business continuity, disaster recovery, resilience, cybersecurity, GRC, telecom and technical writing.</em></p> </section> Growing national and international regulatory compliance demands aim to protect consumer data. Organizations must adhere to regulations or face noncompliance risks. https://cdn.ttgtmedia.com/rms/onlineimages/legal_g1065824400.jpg https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know Wed, 08 Apr 2026 15:31:00 GMT <p>AI and cloud analytics applications are exposing a critical security gap for enterprises. While data is typically secured at rest and in transit, it often remains unprotected when being processed -- the time it is most actively used.</p> <p>This gap has pushed data-in-use protection higher on the agenda for data leaders. Within the broader landscape of privacy‑enhancing technologies, <a href="https://www.techtarget.com/searchsecurity/tip/Confidential-computing-use-cases-that-secure-data-in-use">confidential computing has emerged</a> as the primary way to address this processing‑stage risk. It uses hardware‑isolated trusted execution environments (TEEs) to keep data encrypted during computation, enabling teams to expand AI workloads without overhauling data pipelines or weakening security.</p> <p>Adoption trends suggest confidential computing is moving from a specialized control to a baseline expectation for AI and cloud analytics deployments. In a 2024 report, for example, Grand View Research&nbsp;projected the global market for confidential computing would grow from an estimated $5.46 billion in 2023 to $153.8 billion by 2030, reflecting its increased role as a foundational component of data security.</p> <section class="section main-article-chapter" data-menu-title="How data-in-use protection fits into existing pipelines"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How data-in-use protection fits into existing pipelines</h2> <p>Standard security leaves data vulnerable in system memory and CPUs. Data-in-use protection addresses this exposure problem by keeping information encrypted while workloads execute.</p> <p>At the hardware layer, a <a href="https://www.techtarget.com/searchitoperations/definition/trusted-execution-environment-TEE">TEE</a> is a secure area that runs code and processes data independently from the rest of the system. It isolates data and processing operations to prevent unauthorized access. Even cloud administrators, host OSes and hypervisors do not have access to the data in a TEE.</p> <p>Because confidential computing operates at the infrastructure layer, AI training and analytics jobs can often run in a TEE with minimal architectural changes. TEEs also transparently encrypt processing for applications, minimizing operational disruption while extending protection throughout the compute stage.</p> </section> <section class="section main-article-chapter" data-menu-title="Compliance pressure moves into the processing layer"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Compliance pressure moves into the processing layer</h2> <p>Rapidly evolving regulations are reshaping where organizations invest to secure AI and analytics workloads. A 2025 Stanford report found that <a href="https://www.techtarget.com/searchenterpriseai/feature/AI-regulation-What-businesses-need-to-know">AI-related regulations</a> issued by U.S. federal agencies more than doubled from 25 in 2023 to 59 in 2024. Similarly, the number of AI-related laws passed at the state level increased from 49 to 131.</p> <p>Gartner predicts that by 2029, confidential computing will be used to secure more than 75% of processing operations running in shared infrastructure, such as public cloud services.</p> <p>As sensitive data moves into AI pipelines, the pressure to document security grows. Processing-stage exposure is difficult to control and even harder to record without hardware-based locks. Audit teams and data governance functions that once focused only on storage encryption now require attestation that processing workloads run in protected environments.<br><br>Several regulatory frameworks now explicitly require data-in-use protection:</p> <ul class="default-list"> <li><b>EU AI Act.</b> This new regulation requires documented data governance controls, including evidence of protection during <a href="https://www.techtarget.com/searchenterpriseai/tip/10-steps-to-achieve-AI-implementation-in-your-business">all AI lifecycle stages</a>.</li> <li><b>GDPR.</b> Enforcement of the EU's data privacy regulation is expanding to include data in use, not just in storage or transit.</li> <li><b>PCI DSS v4.0.1.</b> Requirements prevent sensitive authentication data from persisting in memory, such as RAM or memory dumps.</li> <li><b>Digital Operational Resilience Act.</b> DORA mandates data-in-use protection for major EU financial institutions, including controls on data handling within cloud and third‑party processing environments.</li> <li><b>NIST Cybersecurity Framework 2.0.</b> Commonly known as CSF 2.0, it <a target="_blank" href="https://www.nist.gov/cyberframework" rel="noopener">includes</a> data-in-use protection within zero-trust security designs.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="What data leaders gain from data-in-use protection"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What data leaders gain from data-in-use protection</h2> <p>For data leaders, the value of confidential computing aligns with governance, legal and audit functions.</p> <ul class="default-list"> <li><b>Secure access to sensitive data. </b>Healthcare records, financial transaction data, personally identifiable information and other regulated data often aren't used in AI and analytics initiatives due to processing risks. Confidential computing enables access to sensitive data sets without violating governance and security rules.</li> <li><b>Reduced legal exposure</b>. Confidential computing provides verifiable proof that sensitive workloads are processed in hardware-isolated environments, which is especially valuable for documenting regulatory compliance in third-party clouds.</li> <li><b>Increased audit efficiency.</b> The records of secure processing automatically produced by TEEs also reduce manual auditing work and improve audit verification on the use of sensitive data.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Confidential computing use cases"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Confidential computing use cases</h2> <p>The clearest proof of concept for confidential computing comes from highly regulated industries, where organizations face strict requirements around data handling, auditability and cross boundary data sharing.</p> <ul class="default-list"> <li><b>Healthcare. </b>Hospitals and clinical research networks use confidential computing to support federated AI model training across institutions, keeping patient data private in shared systems or central databases.</li> <li><b>Financial services.</b> Banks and insurers use TEEs for fraud detection and risk modeling to reduce exposure when processing sensitive transaction data regulated by banking privacy rules.</li> <li><b>Public sector.</b> Agencies and partner organizations apply confidential computing to joint analytics projects without sharing raw data across organizational boundaries.</li> <li><b>Telecom and IoT.</b> Providers can use confidential computing to analyze customer and device data closer to the edge while limiting exposure during processing.</li> </ul> <p>Across these industries, common use cases include secure AI training, multi‑party analytics, <a href="https://www.techtarget.com/searchenterpriseai/tip/How-to-navigate-data-sovereignty-for-AI-compliance">data sovereignty controls</a>, and cloud backup and recovery workflows where restore operations can expose sensitive data.</p> </section> <section class="section main-article-chapter" data-menu-title="How to evaluate data‑in‑use protection options"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How to evaluate data‑in‑use protection options</h2> <p>Approaches to data‑in‑use protection vary across cloud providers and the broader vendor ecosystem that includes data platforms, security and key management tools, and systems integrators. Before committing to a platform, data leaders should focus on proof points, regulatory alignment and how well it integrates with existing controls.</p> <p><iframe title="" aria-label="Table" id="datawrapper-chart-KuZgD" src="https://datawrapper.dwcdn.net/KuZgD/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="721" data-external="1"></iframe></p> <p> <script type="text/javascript">window.addEventListener("message",function(a){if(void 0!==a.data["datawrapper-height"]){var e=document.querySelectorAll("iframe");for(var t in a.data["datawrapper-height"])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data["datawrapper-height"][t]+"px";r.style.height=d}}});</script> </p> <p><i>Sean Michael Kerner is an IT consultant, technology enthusiast and tinkerer. He has pulled Token Ring, configured NetWare and been known to compile his own Linux kernel. He consults with industry and media organizations on technology issues.</i></p> </section> As sensitive data moves into AI pipelines, organizations must evaluate how to protect it during processing and what safeguards IT platforms provide for data in use. https://cdn.ttgtmedia.com/rms/onlineimages/security_a385093447.jpg https://www.techtarget.com/searchdatamanagement/tip/AI-analytics-push-data-in-use-protection-up-priority-list Tue, 07 Apr 2026 15:34:00 GMT <p>Without properly prepared data, analytics and AI applications are unlikely to deliver the desired business outcomes. But <a href="https://www.techtarget.com/searchbusinessanalytics/definition/data-preparation">data preparation</a> is an inherently complex process that poses various challenges for data management and analytics teams.</p> <p>Preparing data for planned uses requires substantial amounts of time and resources. Indeed, it typically accounts for most of the work involved in developing analytics applications. Large amounts of data in diverse formats <a href="https://www.techtarget.com/searchdatamanagement/feature/Big-data-collection-processes-challenges-and-best-practices">collected from numerous sources</a> must be combined and consolidated. The raw data routinely contains errors, anomalies, inconsistencies and other <a href="https://www.techtarget.com/searchdatamanagement/feature/Proactive-practices-for-data-quality-improvement">data quality issues</a>. Data sets might not include all the information an application requires. Conversely, some data might not be relevant to it.</p> <p>Data preparation tools -- available as separate products or built into BI and data science platforms -- enable data scientists, data engineers, business analysts and other end users to prepare data themselves. However, these tools don't eliminate the challenges of data preparation. Data leaders must ensure users are sufficiently trained on the data prep process, including common challenges.</p> <p>Effective data preparation also requires a multipronged approach. To aid self-service users, data quality analysts <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-cleansing-best-practices">profile and cleanse data</a> upfront. Data integration developers run initial data transformation jobs. BI teams further transform, enrich and curate data sets for planned applications. They, too, must be prepared for the challenges of preparing data.</p> <section class="section main-article-chapter" data-menu-title="7 top data preparation challenges"> <h2 class="section-title"><i class="icon" data-icon="1"></i>7 top data preparation challenges</h2> <p>Because of its complexity, data preparation can't be left to chance. The following are seven notable challenges that disrupt efforts to create clean, consistent and complete data sets, along with advice on how to overcome each one.</p> <h3>1. Inadequate or erroneous data profiling</h3> <p>Data profiling should prevent end users from belatedly discovering data issues when running analytics applications -- or, worse, from having the analytics results be affected by faulty data they aren't aware of. But it might not do so due to the following scenarios:</p> <ul class="default-list"> <li>Data team members or business users preparing data for a new application assume it's valid because it's already used in reports and dashboards. As a result, they don't fully profile the data. However, the existing uses masked underlying problems in the data set.</li> <li>Someone only profiles a sample data set from a large volume of data because of the time it would take to profile the full one. But the sampling approach doesn't detect anomalies and other issues in the full data set.</li> <li>Similarly, custom-coded SQL queries or spreadsheet functions used to profile data aren't comprehensive enough to find all the problems in the data.</li> </ul> <p><b>How to overcome this challenge<br></b>Solid data profiling must be the starting point of the data preparation process. Data preparation tools can help: They include comprehensive functionality for profiling data sets in both source systems and the data platforms that analytics and AI applications run on.</p> <h3>2. Missing or incomplete data</h3> <p>Missing values and incomplete entries are common data quality issues. Examples include:</p> <ul class="default-list"> <li>Null or blank fields.</li> <li>Zeros that represent a missing value rather than the number 0.</li> <li>Other types of placeholder values.</li> <li>Partial transaction records with missing details.</li> <li>Incomplete demographic data on customers.</li> <li>An entire field or row that's missing from a data set.</li> </ul> <p>Missing or incomplete data can adversely affect business decisions driven by analytics applications and create <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-challenges-that-can-sink-data-operations">data governance and regulatory compliance risks</a>. It might also disrupt data loading processes or cause them to fail completely, forcing data teams to scramble to figure out what went wrong.</p> <p>As a result, instances of missing or incomplete data raise complicated data preparation questions. Do they represent substantive data errors? If so, can valid data be inserted? If it can't be, should affected fields be deleted or kept but flagged to show users there are issues with the data?</p> <p><b>How to overcome this challenge<br></b>Effective data profiling identifies missing or incomplete data. Decide what to do about it based on planned use cases and the significance of the data errors. Optimally, data teams or end users should then use a data preparation tool to implement the error-handling measures.</p> <h3>3. Invalid data values</h3> <p>Invalid values are another common data quality issue. They include misspellings, transposed digits, unnecessary characters, duplicate entries and outliers, such as ages, dates and numbers that aren't within a reasonable range. These errors can occur even in enterprise applications with built-in data validation features and end up in analytics and AI data sets.</p> <p>A small number of invalid values in a data set might not have a meaningful impact on applications, but more numerous errors can lead to faulty data analysis results. Cleaning them up should be a priority during data preparation.</p> <p><b>How to overcome this challenge<br></b>Finding and fixing invalid data is similar to handling missing values: Profile the data, decide what to do about errors and implement automated functions to address them. Data profiling should also be done on an ongoing basis to identify new issues as data is updated. Perfection is unlikely -- some data errors inevitably slip through. But minimizing them will prevent bad analytics-driven business decisions.</p> <h3>4. Name and address standardization</h3> <p>Inconsistencies in the names, addresses and contact information of consumers and businesses also complicate data preparation. These are legitimate data variations in different systems, not misspellings or missing values. But if not standardized, they can prevent analytics users and AI tools from getting a complete view of customers, suppliers and other business partners.</p> <p>The following are common examples of such inconsistencies:</p> <ul class="default-list"> <li>A shortened first name or nickname versus a person's full name, such as Fred in one data field and Frederick in another.</li> <li>Middle initial, full middle name or neither.</li> <li>Acronyms vs. full business names, such as BMW and Bayerische Motoren Werke.</li> <li>Companies listed both with and without Inc., Co., Corp., LLC and other business suffixes.</li> <li>Spelled-out vs. abbreviated address data, such as Boulevard and Blvd. or New York and NY.</li> <li>Different phone numbers and email addresses for the same entity.</li> </ul> <p><b>How to overcome this challenge<br></b>Identify inconsistencies through data profiling, then use the standardization features built into a data preparation tool. Alternatively, data teams can create customized standardization processes with a data prep tool's string-handling functionality or use software from a vendor that specializes in name and address standardization.</p> <h3>5. Inconsistent data across enterprise systems</h3> <p>Organizations also encounter inconsistencies when combining data from systems in multiple departments or business units. The data might be correct in each source system, but differences in data formats and entries create problems for analytics and AI applications. It's a pervasive data preparation challenge, especially in large enterprises.</p> <p><b>How to overcome this challenge<br></b>When a data attribute, such as an ID field, has different values across source systems, data conversion or cross-reference mapping procedures provide a relatively easy fix. However, if different business rules or data definitions lead to inconsistencies, more complex&nbsp;data transformations are required.</p> <h3>6. Data enrichment issues</h3> <p>Data enrichment helps create the required business context for effective analytics and AI uses. The following are examples of enrichment measures implemented when preparing data:</p> <ul class="default-list"> <li>Augmenting data with entries from other internal or external sources.</li> <li>Deriving additional data attributes from the existing ones in a data set.</li> <li>Calculating business metrics and KPIs based on the data.</li> <li>Organizing data into different structures for planned applications.</li> <li>Adding tags, labels and metadata to help users understand the data.</li> </ul> <p>But enriching data isn't easy. Deciding what needs to be done is complicated, and enrichment work can be time-consuming.</p> <p><b>How to overcome this challenge<br></b>Data enrichment requires a strong understanding of business needs and goals for the planned applications. Work closely with business executives and users to develop enrichment plans, and allot sufficient resources to the process to meet application delivery schedules.</p> <h3>7. Sustaining and scaling data preparation processes</h3> <p>While data teams and end users sometimes prepare data on an ad hoc basis, data preparation work often becomes a recurring process. Its scope also expands as analytics and AI applications grow and become more widespread -- and valuable -- in enterprises. But organizations often struggle to sustain and scale their data preparation initiatives.</p> <p>Insufficient resources and skills are a problem in some cases. Using custom-coded data preparation methods is, too. If there's no documentation of a custom-coded process, its creator might be the only person who understands how it works, which makes it hard to continue the process if they leave. Also, when modifications to a process are needed, bolting on new code makes maintaining it even more difficult.</p> <p><b>How to overcome this challenge<br></b>Ensure that data preparation programs have the required resources and that data teams and end users are properly trained. Using data preparation tools also helps avoid the traps of custom coding. They automatically document processes and <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-lineage-documentation-imperative-to-data-quality">track data lineage and use</a>, while also providing AI capabilities, collaboration features and connectors to various data sources.</p> <p><b>Editor's note:</b> <i>This article was originally published in 2022. TechTarget editors updated it in March 2026 for timeliness and to add new information.</i></p> <p><i>Rick Sherman, who died in January 2023, was founder and managing partner of Athena Solutions, a BI, data warehousing and data management consulting firm. He had more than 40 years of professional experience in those fields.</i></p> </section> Data preparation is a crucial but complex part of analytics and AI applications. Don't let these seven common challenges send your data prep processes off track. https://cdn.ttgtmedia.com/rms/onlineimages/storage_g539954410.jpg https://www.techtarget.com/searchbusinessanalytics/feature/Top-data-preparation-challenges-and-how-to-overcome-them Tue, 31 Mar 2026 11:00:00 GMT <p>Enterprises are finding the data infrastructure setups that served them well in the past cannot keep up with today's AI reality.</p> <p>A shift from traditional data architectures to a modern data stack is accelerating thanks to an avalanche of AI initiatives -- and a <a href="https://www.techtarget.com/searchenterpriseai/feature/AI-deployments-gone-wrong-The-fallout-and-lessons-learned">lack of trust in the data</a> feeding AI systems. Survey results highlight the problems. Deloitte's 2026 "State of AI in the Enterprise" global survey found that while the number of senior IT and business executives who feel prepared for AI adoption strategically rose to 42 percent from 39 percent the previous year, confidence in their organization's technology infrastructure and data management capabilities declined from 47 percent to 43 percent and from 43 percent to 40 percent, respectively. A 2025 IDC study reported that 84 percent of companies have outdated storage that is not optimal for demanding AI workloads.</p> <p>For enterprise data leaders, it's increasingly a priority to update aging data infrastructure so AI can be deployed with confidence while also modernizing governance and day‑to‑day data management practices that keep AI models reliable and automated decisions defensible.</p> <section class="section main-article-chapter" data-menu-title="From big data complexity to streamlined AI-ready infrastructure"> <h2 class="section-title"><i class="icon" data-icon="1"></i>From big data complexity to streamlined AI-ready infrastructure</h2> <p>The enterprise data stack is evolving out of necessity. To compete in the AI-first economy, organizations are moving <a href="https://www.techtarget.com/searchdatamanagement/opinion/2026-will-be-the-year-data-becomes-truly-intelligent">toward data as a product</a>. This shift replaces brittle, manual workflows with a governed platform designed for scalability, safety and reuse. Under this modern data stack model, IT and data teams provide a secure, shared foundation, while business units maintain ownership of the application outcomes.</p> <p>At each stage of this multilayered approach, data is refined and validated until it is transformed from its raw state into a reusable asset. As organizations roll out autonomous AI agents, this level of granular control over data and <a href="https://www.techtarget.com/searchdatamanagement/post/Key-requirements-for-data-and-analytics-governance-platforms">comprehensive governance</a> is a prerequisite for safe, reliable AI applications at scale.</p> <p>Lists of modern data stack layers aren't standardized, and terminology often differs by the source. However, these are its core elements.</p> <h3>1. Ingestion layer</h3> <p>The first layer covers <a href="https://www.techtarget.com/searchbusinessanalytics/tip/6-essential-big-data-best-practices-for-businesses">data collection</a> and contains the necessary base infrastructure, including compute resources, networking, cloud services and security controls. In traditional data frameworks, this was largely an IT concern, but it is now a strategic design decision upon which the business goals of data-driven applications rest. It's no longer a choice between on-premises and cloud deployments. Instead, data leaders are designing tailored hybrid infrastructures to distribute processing across on-premises systems for data sovereignty, edge locations for real-time AI performance and cloud environments for scalable compute.</p> <p>Teams can use push or pull methods to ingest data from a wide range of internal and external data sources, such as cloud applications and streaming services. In the modern data stack, there is more of a vetting process. Just because vast amounts of data can be ingested into the infrastructure doesn't mean all of it should be. The modern approach also applies a higher bar for data quality, lineage and provenance. The biggest risk in this stage is fragmentation. If data sources remain disconnected, then teams must manually integrate and clean data and redo engineering work, which slows business processes. &nbsp;</p> <h3>2. Storage layer</h3> <p>In traditional data infrastructure, this layer is often a chaotic catch-all. Companies put their ingested raw data in multiple, disconnected databases, which results in conflicting versions of the truth. This legacy approach makes ensuring AI reliability nearly impossible because there is no single, governed source of information. Data warehouses emerged first to consolidate structured data for BI and fast querying. Later, organizations used data lakes to store unprocessed data to support analytics and AI work. However, operating both a data warehouse and data lake creates redundancies with separate systems for storing and managing different data, which adds to governance and security overhead.</p> <p><a href="https://www.techtarget.com/searchdatamanagement/feature/The-differences-between-a-data-warehouse-vs-data-mart">To avoid these data silos</a> in the modern data stack, organizations are now moving to data lakehouses, which combine the cost efficiency of data lakes with the performance of warehouses. The lakehouse architecture enables unified governance by building a metadata layer that oversees both raw and processed data. Also, by using open table formats to build an organization-wide system of record, companies create a consistent foundation for AI model development. This method improves data processing by reducing the need for unnecessary copies of data and manual engineering.</p> <h3>3. Processing layer</h3> <p>This layer turns the raw data into workable assets, ready to be analyzed or fed into AI models. Processing involves preparing both batch data sets at rest and streaming data in motion for downstream analytics and AI use. This data transformation and curation process includes cleansing, standardizing, enriching, filtering, joining and aggregating the data.</p> <p>In the modern data stack, this layer scales beyond the traditional nightly data update cycle designed for BI dashboard environments. The processing layer must handle real-time updates, <a href="https://www.techtarget.com/searchenterpriseai/definition/multimodal-AI">multimodal</a> inputs and automated lineage capture that documents every transformation. This ensures the data's journey from raw to refined is traceable and reduces the risk that AI models will <a href="https://www.techtarget.com/searcherp/podcast/Can-industry-process-models-fix-the-agentic-AI-data-problem">produce hallucinations and other errors</a>. Stream processing enables automated alerts and recommendations to be surfaced as quickly as possible so end users and autonomous agents can take immediate actions.</p> <p>Data leaders should ensure their updated infrastructure can handle this additional work without requiring a patchwork of tools and handoffs, which could create governance gaps.</p> <h3>4. Management and distribution layer</h3> <p>In this layer, the processed data is organized so it is fit for purpose. Built-in features work together not just to make the data available but also to ensure it can be governed and discovered. The work here includes data cataloging, lineage visibility, governance policy enforcement and facilitation of data discovery by downstream users.</p> <p>This is the most critical layer and often determines whether the entire modern data stack succeeds or fails. Ultimately, how most businesses operate today depends on data trustworthiness. Gartner predicts that 50 percent of organizations will use a <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">zero-trust model for data governance</a> by 2028 due to increasing AI adoption. With the growth of AI-generated data, automated data verification and active metadata management in this layer are essential pieces of the zero-trust governance approach.</p> <p>This layer tends to focus on either data mesh or data fabric architectures, each designed to make it easier for users to locate and share data without added complications. A data mesh is built on distributed domain ownership, where different departments are responsible for their own data under a federated governance structure, while a data fabric uses metadata and automated integration capabilities to join divided data assets and make it easier to reuse them.</p> <h3>5. Context and semantic layer</h3> <p>This is the layer where business logic is applied to both refined and raw data, giving it meaning. This context helps end users, AI systems and automation technologies understand how data should be interpreted across the organization.</p> <p>Shared definitions, knowledge graphs, metrics and other structures provide semantic consistency. Connecting context and semantics to data lineage and access policies reduces decision-making time for users and AI tools alike by removing the need to question whether data is relevant to applications. &nbsp;</p> <h3>6. Integrity and quality layer</h3> <p>This layer maintains the fidelity of data as it moves through the stack. It combines data observability, data stewardship, data quality checks and privacy controls to <a href="https://www.techtarget.com/searchdatamanagement/opinion/Data-contracts-help-build-trustworthy-data-products-for-AI">ensure data is accurate</a>, consistent, documented and protected for effective decision-making.</p> <p>This arrangement provides structure to the stack to prevent unreliable data feeds and data silos. Data quality rules identify missing values, data duplication and freshness issues. Master data management practices create common records for business entities, such as customers and products, to maintain consistency across systems. Data stewards apply governance and security policies that <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-challenges-that-can-sink-data-operations">dictate who gets access to data and when</a>.</p> <h3>7. Consumption layer</h3> <p>This is the top of the stack, the culmination of all the architectural choices designed to produce refined, trusted data and get it to the right users and systems at the right time. &nbsp;</p> <p>Traditionally, consumption meant dashboards, reports and analytics tools, but it now includes embedded analytics, machine learning applications, and agentic AI or semi-autonomous workflows. Rather than simply adding AI to old processes, data leaders are redesigning this layer so <a href="https://www.techtarget.com/searchsoftwarequality/tip/How-effective-is-your-AI-agent-benchmarks-to-consider">agents and people can work collaboratively</a> with clear decision-making boundaries, ensuring IT provides the platform while business units determine results.</p> <div class="youtube-iframe-container"> <iframe id="ytplayer-0" src="https://www.youtube.com/embed/7FufIRExfpo?autoplay=0&amp;modestbranding=1&amp;rel=0&amp;widget_referrer=null&amp;enablejsapi=1&amp;origin=https://www.techtarget.com" type="text/html" height="360" width="640" frameborder="0"></iframe> </div> </section> <section class="section main-article-chapter" data-menu-title="What matters most when reassessing the data stack"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What matters most when reassessing the data stack</h2> <p>When it's time to update how your organization processes data and data platform vendors come calling, prepare product evaluation questions to meet your specific needs rather than getting lost in talks about performance and feature checklists.</p> <p>AI initiatives introduce a new set of requirements beyond the capabilities of existing data architectures. Today, the priorities include avoiding data duplication, improved data portability, strong lineage and consistency across departments and clouds.</p> <p>Tailor these modern data stack platform requirements for your organization, but these are some questions to ask:</p> <ul class="default-list"> <li>Does the platform provide a unified semantic layer and active metadata to ensure consistent logic across AI agents and BI applications?</li> <li>Does the platform support hybrid cloud and multi-cloud deployments by design for seamless workload migration based on cost, performance or data sovereignty requirements?</li> <li>Does it have policy-as-code <a target="_blank" href="https://www.cncf.io/blog/2025/07/29/introduction-to-policy-as-code/" rel="noopener">capabilities</a> to standardize data governance, privacy and quality across data assets, and AI models and agents?</li> <li>What are the platform's capabilities related to open table formats, APIs and portable pipelines to avoid extensive work when moving data and workloads?</li> <li>What is the status of agentic AI governance, and what are the plans to close any oversight gaps?</li> <li>Is there a single management interface for data stewards to monitor policy enforcement and issue resolution?</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="What's coming next for the modern data stack?"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What's coming next for the modern data stack?</h2> <p>All signals from leading analyst firms indicate the next evolution of the data stack will refine context awareness, tighten governance and integrate more closely with business workflows and agentic AI systems. These trends are linked: as companies increasingly deploy agents, they need richer context and stronger data controls. Deloitte's 2026 AI survey <a target="_blank" href="https://www.deloitte.com/us/en/what-we-do/capabilities/applied-artificial-intelligence/content/state-of-ai-in-the-enterprise.html" rel="noopener">found</a> that while 74 percent of companies plan to deploy agentic AI within two years, only 21 percent have a governance model for them now.</p> <p>Vendors are converging the stack, joining layers, <a href="https://www.techtarget.com/searchdatamanagement/news/366631576/New-consortium-to-aid-AI-by-standardizing-semantic-modeling">improving semantic structure</a> and embedding oversight. They are moving toward a unified, governed data lakehouse to reduce redundant copies and data movement across silos, cutting costs and security risks. This architecture supports the federated, shared ownership model in which business leaders set standards and quality expectations, while IT manages the data lakehouse and enforces policies to keep data and AI aligned at scale.</p> <p>For organizations reassessing their existing stack architecture, take a modular approach. Avoid overbuying and focus on the immediate needs for data context and trust. This provides flexibility to get AI and analytics work done today rather than a rigid, expensive redesign that might be obsolete in a few years.</p> <p><b>Editor's note</b><i>: TechTarget editors updated this article, originally published in 2023 and written by <a href="https://www.techtarget.com/contributor/Jeff-McCormick">Jeff McCormick</a>, in March 2026 to add new information and improve timeliness.</i></p> <p><i>Tom Walat is an editor and reporter for TechTarget, where he covers data technologies.</i></p> </section> Data infrastructure and practices need an upgrade for the AI era. A modern data stack takes a layered approach that aligns teams and delivers governed, trusted data. https://cdn.ttgtmedia.com/rms/onlineimages/container_g488602622.jpg https://www.techtarget.com/searchdatamanagement/tip/Assemble-the-layers-of-big-data-stack-architecture Fri, 20 Mar 2026 10:24:00 GMT 60 webmaster@techtarget.com