The Growing Challenge of Data Duplication in AI and Analytics

As enterprises increasingly adopt AI-driven analytics, the demand for efficient data access continues to rise. Oracle Autonomous Data Warehouse (ADW) is a powerful platform for analytical workloads, but AI-enhanced processes—such as Agentic AI, Retrieval-Augmented Generation (RAG), and predictive modelling—place new strains on data management strategies.

A key issue in supporting these AI workloads is the need for multiple data copies, which drive up storage costs and operational complexity. Traditional approaches to data replication no longer align with the scale and agility required for modern AI applications, forcing organisations to rethink how they manage, store, and access critical business data.

This blog builds upon my previous post on AI Agents in the Oracle Analytics Ecosystem, further exploring how AI-driven workloads impact traditional data strategies and how organisations can modernise their approach.

​Why AI Workloads Demand More Data

​AI models, particularly those leveraging RAG, generative AI, and deep learning, require constant access to vast amounts of data. In Oracle ADW environments, these workloads often involve:

• Agentic AI and RAG: Continually retrieving and processing real-time or near-real-time data for enhanced decision-making, requiring multiple indexed views of the same dataset.
• Predictive Analytics: Running machine learning models that require extensive historical data for training and inference, often necessitating multiple snapshots of production data.
• Natural Language Processing (NLP): Extracting insights from unstructured data, requiring large-scale indexing, vector search capabilities, and duplication of processed text corpora.
• AI-Driven Data Enrichment: Merging structured and unstructured data sources to generate deeper insights, often leading to multiple temporary and persistent data copies.
• AI Model Testing & Validation: Deploying and fine-tuning AI models across different datasets requires isolated environments, each consuming additional storage resources.

IDC has extensively documented the exponential growth of data and AI investments. Recent industry reports indicate that data storage requirements for AI workloads are expanding at an unprecedented rate.
​
IDC’s broader research reveals several critical insights about AI’s accelerating impact on data ecosystems:
​

1. Global Datasphere Growth: IDC forecasts the global datasphere will reach 394 zettabytes by 2028 (up from 149ZB in 2024), representing a 19.4% compound annual growth rate [11][19]. While this encompasses all data, AI workloads are a primary driver: - 90ZB of data will be generated by IoT devices by 2025, much of it processed by AI systems [2][19].- Real-time data (crucial for AI) will grow to 30% of all data by 2025, up from 15% in 2017[6].
2. AI-Specific Infrastructure Demands - Spending on AI-supporting technologies will reach $337B in 2025, doubling from 2024 levels, with enterprises allocating 67% of this budget to embedding AI into core operations [3][8]. - AI servers and related infrastructure are growing at 29-35% CAGR, outpacing general IT spending [15][17].
3. Generative AI Acceleration IDC predicts Gen AI adoption will drive $1T in productivity gains by 2026, with 35% of enterprises using Gen AI for product development by 2025[4][18]. This requires massive data processing: - Cloud platform services supporting AI workloads are growing at >50% CAGR [5]. - AI-optimised PCs will comprise 60% of all shipments by 2027, enabling localised data processing [20].- Enterprise AI spending doubling from $120B (2022) to $227B (2025) in the U.S. alone[1][3]. - Gen AI spending projected to reach $202B by 2028, representing 32% of total AI investments [8].

The data explosion is being fuelled by AI use cases like augmented customer service (+30% CAGR), fraud detection systems (+35.8% CAGR), and IoT analytics [1][8]. IDC emphasizes that 90% of new enterprise apps will embed AI by 2026, ensuring continued exponential data growth at the intersection of AI adoption and digital transformation [9][12].

AI data volumes are projected to increase significantly, posing challenges for enterprises striving to maintain scalable and cost-efficient storage solutions. Without proactive measures, organisations risk soaring expenses and performance limitations that could stifle innovation.

Sources
[1] Spending on AI Solutions Will Double in the US by 2025, Says IDC https://www.bigdatawire.com/this-just-in/spending-on-ai-solutions-will-double-in-the-us-by-2025-says-idc/
[2] IDC: Expect 175 zettabytes of data worldwide by 2025 - Network World https://www.networkworld.com/article/966746/idc-expect-175-zettabytes-of-data-worldwide-by-2025.html
[3] IDC Unveils 2025 FutureScapes: Worldwide IT Industry Predictions https://www.idc.com/getdoc.jsp?containerId=prUS52691924
[4] IDC Predicts Gen AI-Powered Skills Development Will Drive $1 Trillion in Productivity Gains by 2026 https://www.idc.com/getdoc.jsp?containerId=prMETA51503023
[5] AI consumption to drive enterprise cloud spending spree - CIO Dive https://www.ciodive.com/news/cloud-spend-doubles-generative-ai-platform-services/722830/
[6] Data Age 2025: - Seagate Technology https://www.seagate.com/files/www-content/our-story/trends/files/Seagate-WP-DataAge2025-March-2017.pdf
[7] IDC Predicts Gen AI-Powered Skills Development Will Drive $1 Trillion in Productivity Gains by 2026 https://www.channel-impact.com/idc-predicts-genai-powered-skills-development-will-drive-1-trillion-in-productivity-gains-by-2026/
[8] Worldwide Spending on Artificial Intelligence Forecast to Reach $632 Billion in 2028, According to a New IDC Spending Guide https://www.idc.com/getdoc.jsp?containerId=prUS52530724
[9] Time to Make the AI Pivot: Experimenting Forever Isn’t an Option https://blogs.idc.com/2024/08/23/time-to-make-the-ai-pivot-experimenting-forever-isnt-an-option/
[10] How real-world businesses are transforming with AI - with 50 new stories https://blogs.microsoft.com/blog/2025/02/05/https-blogs-microsoft-com-blog-2024-11-12-how-real-world-businesses-are-transforming-with-ai/
[11] Data growth worldwide 2010-2028 - Statista https://www.statista.com/statistics/871513/worldwide-data-created/
[12] IDC and IBM lists best practices for scaling AI as investments set to double https://www.ibm.com/blog/idc-and-ibm-list-best-practices-for-scaling-ai-as-investments-set-to-double/
[13] Nearly All Big Data Ignored, IDC Says - InformationWeek https://www.informationweek.com/machine-learning-ai/nearly-all-big-data-ignored-idc-says

The Traditional Approach: Cloning Production Data

​Historically, organisations have relied on full database cloning to create isolated environments for AI training, model validation, and analytics. While this approach ensures data consistency, it comes with significant drawbacks:

• Storage Overhead: Each cloned copy requires additional storage, leading to exponential growth in consumption and costs. For organisations processing terabytes or petabytes of data, this rapidly becomes unsustainable.
• Data Staleness: Cloned datasets quickly become outdated, requiring frequent refreshes that consume computing resources and delay AI-driven insights.
• Operational Complexity: Managing multiple cloned copies increases administrative overhead, creating challenges in data governance, version control, and compliance.
• Performance Bottlenecks: As AI models interact with production or cloned datasets, increasing query loads can degrade performance, slowing down analytics and decision-making.
• Security & Compliance Risks: More data copies mean more potential points of exposure, increasing the risk of non-compliance with regulations such as GDPR, CCPA, and industry-specific mandates.

​Cost Implications of Traditional Data Cloning

To put this into perspective, consider a mid-sized enterprise running an Oracle Autonomous Data Warehouse (ADW) instance with 50TB of data. If multiple teams require their own clones for model training and testing, the storage footprint could easily reach 250TB or more. With cloud storage costs averaging £0.02 per GB per month, this could result in annual expenses exceeding £60,000—just for storage alone. Factor in compute, additional database costs and administrative overhead, and the financial impact becomes even more pronounced.

The challenge becomes particularly acute when considering the unique characteristics of AI workloads. Traditional RDBMS architectures were designed for transactional processing and structured analytical queries, but AI workflows introduce several distinct pressures:

Data Transformation Requirements: Machine learning models often require multiple transformations of the same dataset for feature engineering, resulting in numerous intermediate tables and views. These transformations must be stored and versioned, further multiplying storage requirements.

Concurrent Access Patterns: AI training workflows typically involve intensive parallel read operations across large datasets, which can overwhelm traditional buffer pools and I/O subsystems designed for mixed read/write workloads. This often leads to performance degradation for other database users.

Version Control and Reproducibility: ML teams need to maintain multiple versions of datasets for experiment tracking and model reproducibility. Traditional RDBMS systems lack native support for dataset versioning, forcing teams to create full copies or implement complex versioning schemes at the application level.

Query Complexity: AI feature engineering often involves complex transformations that push the boundaries of SQL optimisation. Operations like window functions, recursive CTEs, and large-scale joins can strain query optimisers designed for traditional business intelligence workloads.
​
Resource Isolation: When multiple data science teams share the same RDBMS instance, their resource-intensive operations can interfere with each other and with production workloads. Traditional resource governors and workload management tools may not effectively handle the bursty nature of AI workloads.

Additionally, the need for data freshness adds another layer of complexity. Teams often require recent production data for model training, leading to regular refresh cycles of these large datasets. This creates significant network traffic and puts additional strain on production systems during clone or backup operations.

To address these challenges, organisations are increasingly exploring alternatives such as:
​

1. Data virtualisation and zero-copy cloning technologies
2. Purpose-built ML feature stores with versioning capabilities
3. Hybrid architectures that offload AI workloads to specialised platforms
4. Automated data lifecycle management to control storage costs
5. Implementation of data fabric architectures that provide unified access whilst maintaining physical separation of AI and operational workloads

The financial implications extend beyond direct storage costs. Organisations must consider:
​

• Additional licensing costs for database features required to support AI workflows
• Network egress charges for data movement between environments
• Increased operational complexity and associated staffing costs
• Potential performance impact on production systems
• Compliance and security overhead for managing sensitive data across multiple environments

As AI workloads continue to grow, organisations need to carefully evaluate their data architecture strategy to ensure it can scale sustainably whilst maintaining performance and cost efficiency.

To overcome these challenges, organisations need a solution that optimises storage usage while maintaining seamless access to real-time data. Silk Echo is a powerful tool for optimising database replication in cloud environments. It offers a range of features that improve performance, simplify management, and enhance the resiliency of data infrastructure.

Silk Echo enables virtualised, lightweight data replication. Instead of creating full physical copies of datasets, it provides near-instantaneous, space-efficient snapshots that eliminate unnecessary duplication.

Introducing Silk Echo: A Smarter Approach to AI Data Management

Silk Echo addresses the challenge of data duplication by providing a high-performance virtualised storage layer. Instead of physically copying data into multiple environments, Silk Echo allows AI workloads, data warehouses, and vector databases to operate on a single logical copy. This reduces unnecessary duplication while maintaining high-speed access to data.

How Silk Echo Works

Virtualised Data Access – Silk Echo enables AI workloads to access data stored in Oracle ADW and other environments without requiring full duplication.

High-Performance Caching – Frequently accessed AI data is cached efficiently to provide rapid query performance.

Seamless Integration – Silk Echo integrates with Oracle ADW, vector databases, and AI model pipelines, reducing the need for repeated ETL processes.

Cost Optimisation – By eliminating redundant data copies, organisations can significantly cut down on storage costs while maintaining AI performance.
​
Silk Echo represents a shift in how enterprises approach AI and data management, ensuring that AI workloads remain cost-efficient, scalable, and manageable within Oracle ADW environments. The next step is to explore how Silk Echo integrates with specific Oracle AI use cases.

Key Benefits of Silk Echo for Oracle ADW and AI Workloads

Products like Silk’s Echo offering, provide a number of benefits to the RDBMS architecture that enable the efficient cost-effective support of modern AI workloads. Some of these benefits are:
​

• Storage Optimisation: Eliminates redundant data copies, reducing storage consumption by up to 80% and significantly lowering costs.
• Real-Time Data Access: Ensures AI models always work with the most up-to-date information, reducing the lag introduced by traditional cloning processes.
• Accelerated AI & Analytics Workflows: Removes bottlenecks associated with traditional cloning, improving overall data pipeline efficiency.
• Enhanced Data Governance & Security: Reduces data sprawl, helping organisations maintain compliance and security standards with minimal administrative burden.
• Faster AI Model Development & Deployment: Enables AI teams to test and validate models with up-to-date snapshots instead of relying on costly, static cloned environments.

​Future-Proofing Oracle ADW and Oracle Analytics for AI Workloads

The rapid evolution of AI and analytics demands that organisations build future-proof architectures that can scale with new workloads. Silk Echo plays a crucial role in this by:

• Enabling AI-Ready Data Architectures: With Silk Echo, Oracle ADW can handle the increasing demands of AI-driven analytics without compromising performance or cost efficiency.
• Supporting AI Innovations: As AI models become more sophisticated, they will require dynamic and optimised access to real-time data. Silk Echo ensures that models always have the freshest data available.
• Ensuring Long-Term Cost Efficiency: By minimising unnecessary data replication, Silk Echo provides a sustainable cost model that allows organisations to allocate resources more effectively to AI initiatives.
• Enhancing Data Virtualisation Capabilities: The ability to create lightweight, instant extracts means organisations can easily integrate Oracle Analytics with broader AI ecosystems, improving analytical outcomes.

The Future of AI and Analytics in Oracle ADW

As AI adoption grows, businesses must rethink their data strategies to balance performance, cost, and scalability. By leveraging Silk Echo in Oracle ADW environments, organisations can:

• Reduce the financial burden of storage-intensive AI processes.
• Ensure AI-driven applications operate with real-time, accurate data.
• Improve compliance and governance without slowing down innovation.
• Scale AI and analytics workloads without excessive data duplication.

Are You Ready to Optimise Your AI-Driven Analytics in Oracle ADW?

By adopting next-generation storage solutions like Silk Echo, organisations can unlock the full potential of AI while keeping costs under control. Investing in efficient data management strategies today will ensure businesses remain competitive in the AI-driven future.

The rise of Agentic AI is transforming the analytics landscape, but it comes with an often-overlooked challenge: database strain. Traditionally, operational databases are ringfenced to prevent unstructured, inefficient queries from affecting critical business functions. However, in a world where AI agents dynamically generate and execute SQL queries to retrieve real-time data, production databases are facing unprecedented pressure.
​
Additionally, Retrieval-Augmented Generation (RAG), a rapidly emerging AI technique that enhances responses with real-time data, is further intensifying this issue by demanding continuous access to up-to-date information. RAG works by supplementing AI-generated responses with live or external knowledge sources, requiring frequent, real-time queries to ensure accuracy. This puts even more strain on traditional database infrastructures. In a previous blog post, I looked at how Agentic AI will improve the experience for users of the Oracle Analytics ecosystem.

The Problem: AI Agents, RAG & Uncontrolled Query Load

A well-managed production database is typically shielded from unpredictable query loads. Database administrators ensure that only structured, optimised workloads access production systems to avoid performance degradation. But with Agentic AI and RAG, that fundamental principle is breaking down.
Instead of a few human analysts running queries, organisations may now have dozens or even hundreds of AI agents autonomously executing SQL queries in real time. These queries are often:

• Ad hoc and unpredictable, making performance tuning difficult
• Highly frequent, since AI agents are designed to work autonomously
• Complex and computationally expensive, often scanning large datasets

This creates significant challenges for traditional RDBMS architectures, which were not designed to handle the scale and unpredictability of AI-driven workloads. With Retrieval-Augmented Generation (RAG) in particular, AI models require frequent access to real-time data to enhance their outputs, placing additional stress on transactional databases. Since these databases were optimised for structured queries and controlled access, the introduction of AI-driven workloads risks causing slowdowns, performance degradation, and even system failures.

For users of Oracle Analytics, this shift presents serious performance implications. If production databases are overwhelmed by AI-driven queries, query response times increase, dashboards lag, and real-time insights become unreliable. Additionally, Oracle Analytics’ AI Assistant, Contextual Insights, and Auto Insights features, which rely on efficient access to data sources, could suffer from delays or inaccuracies due to excessive load on transactional systems.

To mitigate this, organisations must rethink their database strategies, ensuring that AI workloads are governed, optimised, and properly distributed across more scalable architectures.

The Traditional Approach: Cloning Production Data

One way that organisations have attempted to address this issue is by cloning production databases on a daily or weekly basis to offload AI-driven queries. However, this approach presents several major drawbacks:

• Clones are not real-time – Since they are refreshed periodically, they lack the latest data required for up-to-the-minute AI-driven insights.
• Cloud clones are expensive – Storage and compute costs for cloud-based clones can be prohibitively high, making large-scale adoption unsustainable.
• Complexity & Latency – Maintaining and synchronising clones across multiple environments increases operational overhead and slows down analytics workflows.
• Doesn’t scale with RAG demands – Since RAG requires real-time retrieval of data to generate accurate responses, stale clones fail to meet its needs.

For AI-driven analytics, relying on periodic database clones leads to significant inefficiencies. Since AI agents require access to up-to-date, contextual information, working with outdated data introduces data integrity risks and lowers the effectiveness of AI-generated insights. Additionally, managing and maintaining multiple clones across different environments adds a significant administrative burden, requiring additional governance, access control, and monitoring.

For Oracle Analytics users, these challenges could lead to outdated insights, reduced trust in AI-generated recommendations, and a poor user experience due to lagging or inconsistent data. Given these drawbacks, it’s clear that cloning is not a viable long-term solution for handling the database demands of Agentic AI.

A Shift in Data Architecture: Data Lakes & Lakehouses

Instead of relying on traditional RDBMS architectures, organisations are increasingly adopting data lakes and lakehouses to support AI-driven analytics. These architectures offer several key advantages:

• Decoupled storage and compute – Unlike traditional databases, lakehouses allow AI agents to access structured and unstructured data without directly querying transactional systems.
• Scalability – Cloud-based lakehouses scale seamlessly with AI workloads, reducing the risk of database bottlenecks.
• Cost-effectiveness – By storing data in low-cost cloud object storage, lakehouses significantly reduce the expenses associated with cloning full databases.

However, while lakehouses present an effective alternative, the migration from a traditional RDBMS to a lakehouse architecture is not without its challenges. The shift requires significant investment in re-architecting data pipelines, ensuring data governance, and retraining teams to work with new query engines and tools. Moreover, performance tuning for structured transactional data in a lakehouse can be complex compared to optimised RDBMS queries, which may lead to initial adoption friction.
​
For users of Oracle Analytics, this shift could mean that existing reports and dashboards need to be refactored to work efficiently with a lakehouse structure, adding additional effort and complexity.

Optimising Performance with Modern Storage Solutions

Beyond adopting new architectural patterns, organisations can leverage modern storage solutions like Silk to mitigate the strain on production databases. Silk provides a virtualised, high-performance data layer that optimises storage performance and scalability without requiring a complete architectural overhaul.
By using Silk or similar intelligent storage virtualisation and caching technologies, organisations can:

• Provide instant, efficient data replication – Unlike traditional cloud-based clones, modern solutions can create real-time, low-overhead replicas, ensuring fresh data access without excessive costs.
• Optimise query performance – Intelligent caching and virtualisation technologies help ensure AI workloads do not compromise the performance of mission-critical systems.
• Reduce the complexity of migrating from an RDBMS – By providing compatibility with existing RDBMS environments, these solutions offer a more gradual and manageable transition to modern data architectures.

For organisations using Oracle Analytics, integrating such solutions could help sustain real-time data access while alleviating the performance burden on production databases. However, despite these advantages, storage virtualisation and caching solutions are not a panacea. Organisations must still ensure that their AI workloads are properly governed to prevent excessive resource consumption, and they need to assess whether virtualised storage aligns with their broader data architecture and security policies.

Conclusion: Preparing for the Future of AI-Driven Analytics

Agentic AI and RAG are here to stay, and with them comes a fundamental shift in how data is accessed and managed. However, blindly allowing AI-driven queries to run against production databases is not a sustainable solution. To support the evolving demands of AI, organisations must modernise their data strategies by:

• Shifting AI workloads to scalable architectures like data lakehouses
• Implementing AI query governance to optimise performance
• Leveraging modern storage technologies like Silk to mitigate traditional RDBMS bottlenecks

For Oracle Analytics users, this shift will require rethinking how data is stored, accessed, and processed to ensure that the platform continues to deliver timely insights without compromising performance. The key takeaway? Traditional database architectures were not designed for AI-driven workloads. To fully embrace the potential of Agentic AI and RAG, organisations must rethink their data foundations - or risk being left behind.

How is your organisation adapting to the challenges of AI-driven analytics? Let’s continue the conversation in the comments!

How AI is Reducing SQL’s Visibility in Oracle Analytics

AI-powered features in Oracle Analytics allow users to explore data without manually writing SQL. Three key capabilities demonstrate this shift:

1. Contextual Insights: Auto-Generated SQL Behind the Scenes
• Example: A sales manager sees an unexpected spike in revenue on a dashboard. Instead of running queries, they click on the data point, and Contextual Insights automatically surfaces key drivers and trends.
• What happens in the background? Oracle Analytics generates SQL queries to identify correlations, anomalies, and patterns, but the user never sees or writes them.

2. AI Assistant: Querying Data Without SQL
• Example: A marketing analyst wants to compare Q1 and Q2 campaign performance. Instead of writing a SQL query, they ask the AI Assistant:
“Show me campaign revenue for Q1 vs Q2.”
• What happens in the background? The AI Assistant translates the request into SQL, retrieves the data, and presents a visual answer.
• Why it matters: Business users get answers instantly, without needing SQL expertise.

3. Auto Insights: Surfacing Trends Without Querying Data
• Example: A finance team wants to understand profit fluctuations. Instead of manually querying revenue data over time, they use Auto Insights, which highlights key trends and anomalies.
• What happens in the background? Oracle Analytics runs SQL queries to detect significant changes and patterns.
​
These features make SQL less visible but not obsolete. In fact, AI relies on SQL to function effectively, which leads to the question—where is SQL still essential?

Why SQL is Still Essential

While AI is making SQL more accessible, it hasn’t eliminated the need for SQL expertise. Several areas still require manual intervention:

1. Handling Complex Joins & Business Logic
• AI struggles with complex queries that involve multiple joins, subqueries, and conditional logic.
• Example: A financial analyst wants to calculate profitability by region, requiring a multi-table join across sales, inventory, and expenses. AI might generate an inefficient or incorrect query.

2. Performance Optimisation
• AI-generated SQL isn’t always the most efficient. SQL tuning (e.g., indexing, partitioning) still requires human expertise.
• Example: AI might generate a query that performs a full table scan instead of leveraging an index, slowing down performance.
​
3. Explainability & Trust
• AI-generated queries can sometimes produce unexpected results, making it difficult for users to validate the logic.
• Example: If AI Assistant returns an unusual data trend, an analyst may still need to inspect the underlying SQL to ensure accuracy.

SQL remains a crucial tool for data engineers, analysts, and DBAs who need control over data processing, query performance, and governance. However, as AI continues to evolve, could it overcome these challenges?

The Role of the Semantic Model in AI-Driven Analytics

One of the key features of Oracle Analytics is its semantic model, designed to abstract the complexity of source systems from end users. Instead of writing raw SQL queries against complex database structures, users interact with a logical layer that simplifies relationships, calculations, and security rules.

Why the Semantic Model Exists

The semantic model serves several purposes, including:

• Hierarchies & Drilldowns: Defining business hierarchies (e.g., Year → Quarter → Month) for intuitive analysis.
• Logical Joins & Business Logic: Providing a structured way to join tables without requiring users to understand foreign keys or database relationships.
• Row-Level Security: Enforcing access control so users only see the data they are authorised to view.

This abstraction enables self-service analytics while ensuring data governance, performance, and accuracy.
Will AI Make the Semantic Model Redundant?

AI-powered analytics features in Oracle Analytics Cloud (OAC)—such as Contextual Insights, AI Assistant, and Auto Insights—are reducing the need for manual query writing. But does this mean the semantic model is no longer needed? Not quite.

AI currently relies on the semantic model to:

• Ensure accurate and governed data access—AI cannot enforce security rules or business logic without a structured data layer.
• Interpret user queries correctly—When an AI Assistant generates SQL, it uses predefined joins and relationships from the semantic model.
• Maintain consistency—Without a semantic layer, different AI-generated queries might return inconsistent results due to varying assumptions about data relationships.

The Future: AI-Augmented Semantic Models?
​
Rather than replacing the semantic model, AI could enhance it by:

• Auto-generating relationships & joins based on data patterns.
• Improving performance optimisation, recommending indexing strategies or pre-aggregations.
• Enhancing explainability, showing why certain joins or hierarchies were applied.

AI and the Semantic Model Will Coexist

While AI reduces the need for manual SQL, the semantic model remains essential for structured, governed, and performant analytics. The future is likely AI-assisted semantic models rather than their elimination.
The Future of AI in SQL GenerationAI will likely become more sophisticated in handling SQL, but rather than eliminating it, AI will enhance SQL’s role. Here’s what the future might look like:

1. AI-Powered Query Optimisation
• AI could not only generate SQL but also analyse and optimise it for better performance.
• Example: Future AI Assistants might suggest indexing strategies, rewrite inefficient queries, or recommend materialised views.
2. Better Handling of Complex Joins & Business Logic
• AI could integrate knowledge graphs or semantic layers to better understand relationships between tables, improving the accuracy of generated SQL.
3. Explainable AI for SQL Generation
• AI might offer query rationale explanations, showing users why a specific query was generated and suggesting alternative approaches.
4. AI Agents & Autonomous Databases
• AI Agents could work alongside SQL experts, automating routine queries while letting humans handle complex cases.
• Oracle’s Autonomous Database could play a larger role in self-optimising SQL execution.
​While AI will continue to reduce the need for manual SQL writing, it is more likely to enhance SQL rather than replace it.

As artificial intelligence redefines enterprise technology, this blog explores the cutting-edge potential of AI Agents within Oracle Analytics Cloud - presenting a visionary look at how intelligent, autonomous systems will transform data analytics and decision-making. 
​
AI agents have emerged as transformative tools in the realm of analytics and are quickly becoming the "new talk of the town" in the tech world. These advanced systems, driven by large language models (LLMs) and machine learning, are at the forefront of the next wave in generative AI.

​For instance, OpenAI's ChatGPT now includes plugins and tool integrations that enable it to function as an AI agent capable of completing tasks like booking appointments, analysing datasets, or generating tailored recommendations. Similarly, Google's Gemini incorporates tool usage and contextual learning, making it highly adaptable to user needs. AI agents can autonomously perform tasks, analyse data, and provide actionable insights, representing a shift towards more interactive and intelligent systems.

With leading LLMs introducing AI agent capabilities, it’s worth exploring how these agents can benefit Oracle Analytics Cloud (OAC) and what use cases they can unlock.

What Are AI Agents?

​AI agents are intelligent systems designed to autonomously execute tasks, solve complex problems, and make informed decisions by leveraging their programming and available data. At a high level, they operate by combining three core capabilities:

1. Perception: Using natural language processing (NLP), AI agents can understand user inputs in conversational language, enabling seamless interaction.
   
2. Reasoning: AI agents employ machine learning (ML) and logic-based algorithms to analyse inputs, make decisions, and draw meaningful conclusions based on the context and data available.
   
3. Action: They execute tasks or deliver insights based on their reasoning, often taking proactive steps such as sending alerts, generating reports, or recommending next actions.
   

What sets AI agents apart from traditional systems is their ability to continuously learn from interactions, dynamically adapt to new information, and simulate human-like understanding. In essence, they act as highly capable intermediaries, bridging the gap between user intent and actionable insights. They can:

• Interpret user input in natural language.
  
• Access multiple data sources.
  
• Automate workflows and decision-making.
  
• Continuously learn and adapt to new information.
  

In analytics, AI agents amplify user productivity by reducing manual intervention and enabling data-driven decision-making at scale.

Oracle Digital Assistant: An Example of Agentic AI

Oracle Digital Assistant (ODA) is a prime example of how Oracle has already embraced agentic AI. ODA combines conversational AI and task automation to enable businesses to build interactive, intelligent chatbots.

While traditional chatbots often follow predefined workflows with limited intelligence, ODA incorporates machine learning and NLP to adapt to user queries dynamically. It can integrate with Oracle applications, providing users with personalised recommendations, automating repetitive tasks, and enhancing the overall user experience.

By acting as a virtual assistant that understands context and intent, ODA showcases how Oracle has been leveraging AI agent-like capabilities to enhance enterprise productivity.

Benefits of AI Agents in Oracle Analytics Cloud

OAC already offers powerful AI and machine learning features, such as natural language querying, auto-insights, and contextual insights. Integrating AI agents with OAC can enhance these capabilities by:

1. Streamlining User Interaction: AI agents can serve as virtual assistants, interpreting user queries and providing insights without the need for complex dashboard navigation. This aligns with OAC’s mission to simplify data interaction for users of all skill levels, making self-service analytics more accessible and efficient.
2. Automating Repetitive Tasks: From generating reports to setting alerts, AI agents can automate routine analytics processes, saving time and resources. By reducing the manual effort involved in tasks, users can focus more on strategic decision-making rather than operational details.
3. Proactive Decision-Making: With real-time data access and advanced reasoning, AI agents can alert users to anomalies, trends, or opportunities proactively. This ensures businesses can respond swiftly to changing conditions, enhancing agility and competitiveness.
4. Personalised Insights: AI agents can tailor insights to individual user needs, learning preferences over time. This personalisation empowers users by delivering relevant, actionable data, which is critical in decision-making.
5. Improved Accessibility: By enabling natural language interaction, AI agents lower the barrier to entry for non-technical users, fostering broader adoption of analytics tools. This supports OAC’s commitment to extending self-service analytics capabilities to a wider audience and keeps Oracle at the forefront of the analytics market by addressing the growing demand for intuitive and accessible solutions.
   

Existing AI Agent Functionality in OAC

​OAC already incorporates several AI-driven features that mimic AI agent functionality, including capabilities that proactively assist users, streamline analytics tasks, and enhance decision-making. These features leverage automation, machine learning, and natural language processing to deliver intelligent insights and actionable recommendations, much like a fully realised AI agent would.​
​

• Natural Language Querying (NLQ): Allows users to ask questions like “What are my top 5 customers by revenue?” and receive instant visualisations.
• Auto Insights: Automatically identifies patterns, anomalies, and trends in data.
• Contextual Insights: Offers intelligent recommendations and additional context based on the data being analysed.
• Machine Learning Models: Empowers users to build and deploy predictive models without extensive technical knowledge.

Conclusion

​The integration of AI agents into Oracle Analytics Cloud represents a transformative leap forward in enterprise analytics. By seamlessly blending autonomous intelligence with data analysis, OAC is poised to redefine how organisations extract, interpret, and act on insights. These AI agents will not merely enhance current workflows—they will fundamentally reimagine the analytics experience, enabling predictive, proactive, and personalised decision-making at an unprecedented scale.

As AI continues to evolve, Oracle Analytics Cloud stands at the forefront of a paradigm shift. While competitors like Microsoft Power BI and Tableau are exploring AI-driven features, OAC has the potential to leapfrog traditional approaches by embedding truly intelligent, contextually aware agents that can autonomously navigate complex data landscapes. The future of analytics is not just about reporting—it's about creating intelligent systems that anticipate needs, generate insights, and drive strategic actions with minimal human intervention
.
The journey of AI agents in Oracle Analytics Cloud is just beginning, promising a new era of data intelligence that transforms how organisations understand and leverage their most critical asset: information.

As organisations strive to make faster, smarter decisions, analytics tools must evolve to offer more than static dashboards and manual data exploration. Enter Contextual Insights, a game-changing feature set to debut in Oracle Analytics Cloud (OAC) as part of the January 2025 update.

Thanks to the Oracle Analytics Product Management team, I was given early access to this feature and have had the opportunity to explore it hands-on. In this blog, I’ll share my insights, experience, and feedback to help you understand the transformative potential of Contextual Insights.

What Are Contextual Insights

Contextual Insights are dynamically generated insights that appear based on the context of the data being analysed. Powered by Oracle’s advanced AI algorithms, these insights surface anomalies, trends, and patterns without requiring users to manually search for them.

For instance, imagine you’re analysing sales data for a retail chain. Contextual Insights could highlight a sudden drop in sales for a specific region or a spike in returns for a particular product category—all without you needing to ask.

Key Features of Contextual Insights

    1.    AI-Powered Recommendations
Contextual Insights leverage AI to suggest deeper analysis opportunities, such as exploring correlations between variables or detecting unusual behaviour in your data.
    2.    Dynamic Visualisations
The insights are not just textual; they include visual representations such as trend lines, scatter plots, or bar charts, making the findings easier to understand at a glance.
    3.    User-Centric Design
These insights are tailored to the user’s role and the context of their query, ensuring that the most relevant information is surfaced.
    4.    Seamless Integration
Contextual Insights work seamlessly with other OAC features like Auto Insights, Narratives, and the Natural Language Query (NLQ) interface.

How Contextual Insights Fit Into OAC​

Oracle has designed Contextual Insights to seamlessly integrate into the Oracle Analytics Cloud (OAC) experience, enhancing usability while preserving the intuitive workflows users rely on. This integration ensures that insights appear naturally during the analytical process, enriching user interactions without introducing complexity.

Key Examples of Integration:
​
    •    Enhanced Visualisation Analysis:
When users view a visualisation, Contextual Insights proactively surfaces anomalies, outliers, or unexpected trends specific to the data currently in focus. For instance, a sales trend chart might highlight a sudden dip in revenue for a specific region or product line, prompting immediate investigation.
    •    Ad-Hoc Analysis Empowerment:
In exploratory scenarios, where users are conducting ad-hoc analysis, Contextual Insights helps uncover patterns or correlations that might not have been apparent. For example, when analysing marketing campaign data, it might reveal that a spike in customer engagement correlates with specific demographic segments or external factors.

A Unified Workflow for Enhanced Decision-Making:

This tight integration allows Contextual Insights to complement the natural flow of analysis, ensuring that users discover actionable insights without needing to disrupt their existing processes or switch contexts. By blending seamlessly with tools like dashboards, visualisations, and exploration interfaces, Contextual Insights empowers users to focus on making data-driven decisions rather than spending time searching for information.

Bridging Technical Gaps:

For non-technical users, this integration is particularly valuable, as it eliminates the need for advanced data expertise to identify meaningful insights. Meanwhile, for advanced analysts, it acts as a catalyst, streamlining deeper exploration and enabling quicker hypothesis testing.

By embedding Contextual Insights deeply within OAC, Oracle delivers an analytics experience that is not only smarter but also more intuitive and inclusive, ensuring users at all levels can uncover hidden opportunities with ease.

Why Contextual Insights Matter

In a world where data-driven decisions are becoming the norm, the ability to uncover relevant, actionable insights at the right moment can be the difference between staying ahead of the curve and falling behind.
1. Democratisation of Analytics
Contextual Insights empower users who may not have technical expertise in data analytics, widening the reach of OAC across organisations.

2. Enhanced Productivity

By surfacing insights automatically, analysts save time previously spent on manual data exploration.

3. Proactive Decision-Making
Contextual Insights shift the focus from reactive reporting to proactive planning by identifying trends and anomalies in real-time.

How Contextual Insights Complements Auto Insights
While Auto Insights in Oracle Analytics Cloud focuses on providing users with automatically generated, high-level summaries and narratives about their data, 

Contextual Insights takes this a step further by tailoring those insights to the user’s specific context. Auto Insights excels at offering a broad overview, such as key performance indicators or summarised trends, whereas Contextual Insights dynamically surface patterns, anomalies, and trends based on the user’s immediate data interactions. Together, these features create a seamless experience where users can move from a high-level understanding to in-depth exploration, uncovering actionable insights with minimal effort. This synergy ensures that users at all skill levels can maximise the value of their data, moving from descriptive to diagnostic analytics effortlessly

Real-World Use Cases

Contextual Insights unlock powerful opportunities across a variety of business sectors by surfacing patterns, trends, and anomalies that have the potential to drive more informed and proactive decision-making.

1. Retail: Identify unexpected dips in regional sales and explore correlations with weather patterns or competitor activity.
   
2. Finance: Detect anomalies in expense reports that could signal fraud or misallocation.
   
3. Supply Chain: Forecast seasonal demand fluctuations to optimise inventory levels.
   
4. Healthcare: Spot anomalies in patient outcomes across different departments or treatments, enabling targeted improvements in care delivery.
   
5. Manufacturing: Detect production delays or quality issues in real-time, with insights into root causes such as supply chain disruptions or equipment malfunctions.
   
6. Public Sector: Monitor trends in service delivery, such as identifying underserved communities or tracking anomalies in public funding allocations.

How to Get Started

Once the January 2025 update is live, enabling and using Contextual Insights will be straightforward. 

    1.    It is configured at a visualisation level.
    2.    Ensure that Contextual Insights is enabled for your visualisations. .
    3.    Select the data item to analyse and select the "Explain Selected" option from the context menu. 
​
For more detailed steps, refer to Oracle’s YouTube videos below explaining the feature and also detailing the configuration steps to set up Contextual Insights. 

Final Thoughts

​Contextual Insights represent a major leap forward in empowering organisations to make faster, smarter, and more informed decisions. By integrating advanced AI-driven capabilities directly into Oracle Analytics Cloud, this feature enables users of all skill levels to uncover hidden opportunities and respond proactively to emerging trends.

As analytics tools evolve, features like Contextual Insights showcase Oracle’s commitment to democratising analytics and fostering innovation. Whether you’re a data novice or a seasoned data scientist, Contextual Insights can transform how you explore and act on your data.

Embrace this feature to unlock the full potential of your analytics workflows and drive meaningful outcomes in your organisation. If you’re ready to explore its possibilities or need support with OAC, reach out or leave a comment below!

In today’s data-driven world, the ability to transform unstructured data into actionable insights is critical for organisations. With the November 2024 release of Oracle Analytics Cloud (OAC), the integration of OCI Document Understanding that brings cutting-edge capabilities to businesses looking to unlock the value hidden in their documents has been extended to allow users to register custom models. You can find out more information on how to create a custom model here.

In this blog, we will be looking at document understanding and how it fits into analytics. Here’s how this feature empowers analytics workflows.

From Unstructured to Actionable: The Role of Text Extraction

Many essential business processes rely on unstructured documents such as contracts, invoices, shipping manifests, and feedback forms. These documents often contain vital data, but their formats - PDFs, scanned images, or handwritten forms - make extracting and analysing this data manually a time-consuming and error-prone process.

Key Benefits of Text Extraction for Analytics​

​Text extraction is often viewed as a preliminary step rather than an intrinsic part of analytics, but this perspective underestimates its transformative impact on modern data workflows. In today’s organisations, vast amounts of critical information remain trapped in unstructured formats - documents, emails, contracts, and scanned images. Without the ability to extract and structure this data, analytics initiatives risk missing out on valuable insights hidden in plain sight like having a collection of . By integrating text extraction directly into analytics workflows, businesses not only bridge the gap between unstructured and structured data but also enhance the scope and accuracy of their insights.

While it may seem that text extraction belongs solely to the domain of data preparation, its seamless integration into analytics platforms changes the game. By enabling users to work directly with previously inaccessible information, text extraction ensures that analytics becomes truly comprehensive. This convergence eliminates the need for siloed processes, accelerates decision-making, and empowers users to leverage their data assets fully. As the lines between data preparation and analytics blur, text extraction proves itself not as a separate utility but as an essential enabler of meaningful, end-to-end analytics workflows.

Some of the benefits of integrating text extraction with analytics are:

1. Streamlined Data Preparation

Extracted text is ready for analysis without requiring extensive manual intervention. For example, a retail company can process thousands of supplier invoices, extracting line-item details such as product names, prices, and quantities. This structured data feeds into Oracle Analytics for further preparation and enrichment, such as cleansing inconsistent naming conventions or enriching data with external sources.

2. Improved Decision-Making

By leveraging the extracted text, users can create dashboards that provide actionable insights. A logistics company, for example, might track delivery times and costs across suppliers, identifying inefficiencies and opportunities to renegotiate contracts.

3. Cross-Document Analysis

OCI Document Understanding enables businesses to analyse trends across a corpus of documents. A financial institution can aggregate key metrics from thousands of contracts, such as interest rates or repayment terms, to assess portfolio risk and optimise lending strategies.

4. Advanced Search and Contextual Insights

Once text is extracted, it can be indexed and searched, enabling users to locate specific terms or patterns across document sets. For instance, legal teams can identify clauses that might expose the organisation to risk, while sales teams can quickly review terms in customer contracts to tailor offers.

Registering a Pre-trained Document Key Value Extraction Model in Oracle Analytics Cloud

Oracle Analytics Cloud provides access to some pre trained OCI document understanding models. This process allows you to leverage the AI capabilities of OCI Document Understanding within OAC to automatically extract key data points from your documents. Here are the detailed steps involved:

Access the Model Registration Function: Begin by navigating to the OAC Home Page. In the top right corner, locate the three-dot menu (ellipsis) and select "Register Model/Function." From the options presented, choose "OCI Document Understanding Models"

Establish the OCI Connection: Next, you'll need to select your OCI connection. If you haven't already established a connection between OAC and OCI, you'll be prompted to create one. This connection is crucial as it enables OAC to interact with the OCI Document Understanding service. 

Select the Desired Model Type: Once the OCI connection is established, a "Select a Model" window will appear. Choose "Pretrained Document Key Value Extraction" as the model type. This specific model is designed to identify and extract key data from documents, such as merchant names, addresses, and total prices.

Specify the OCI Bucket and Document Type: In the right-side panel of the "Select a Model" window, you'll need to provide two crucial pieces of information:

1. OCI Bucket: Select the OCI bucket where you have stored the document images you want to analyze. Ensure this bucket is located within the same tenancy as your OAC instance2.○
2. Document Type: Choose the specific document type that corresponds to the images in your bucket. For instance, if you are analysing receipts, select "Receipt." This helps the model accurately identify and extract the relevant key values for that particular document type.

Provide a Model Name and Register: Finally, give your model a descriptive name for easy identification within OAC. Click "Register" to complete the process. You can view your registered model under the "Models" tab in the Machine Learning page of OAC. By following these steps, you successfully register a pre-trained document key value extraction model in OAC, setting the stage for streamlined data preparation and enhanced data analysis. You can then create data flows within OAC to apply this registered model to your documents, extract the desired key values, and use this structured data to generate valuable insights. 

Oracle Analytics have just announced the general availability of the AI Assistant. It is a new feature that allows users to interact with their data using natural language. This means you can ask questions in plain English and the assistant will generate visualisations and insights based on your dataset.

Democratisation of Data Analytics: The AI assistant makes data analysis accessible to a wider range of users, not just those with advanced technical skills or even those with a formidable understanding of the data visualisation tool. By allowing users to interact with data using natural language, it eliminates the need for complex SQL queries or programming knowledge. This democratisation empowers more people within an organisation to explore data and derive valuable insights.

This feature is part of Oracle’s broader AI-driven efforts within Oracle Analytics Cloud, which also includes integration with machine learning models and advanced natural language generation for smart data narratives

Below, is a summary of the steps to follow to get the AI Assistant up and running:
​

• Enable your dataset: Ensure your dataset is ready for use by checking the Insights panel and enabling it if necessary.
• Train the assistant: Index your dataset to help the assistant understand your data and answer questions accurately.
• Ask questions: Start asking questions using natural language. The assistant will generate visualizations and provide insights based on your data.

This Oracle Analytics blog provides detailed information of these simple steps needed to set this up. It requires some configuration to enable the feature at a dataset level. To improve the user experience, setting up synonyms for the dataset attributes is useful as this gives you the option to provide alternative names for each attribute which will improve the results generated by the AI Assistant. 

Remember to choose the option to index the dataset for the Assistant. There is also an option to index the dataset for the Assistant and the Homepage Ask feature if required. 

Upon initial review, the Oracle Analytics AI Assistant shows significant potential for enhancing productivity and easing the workflow, particularly for those less familiar with advanced analytics tools. While the setup, particularly dataset configuration and indexing, may require some upfront effort, once configured, the tool truly shines. The integration of natural language querying makes it much easier to interact with the data, reducing the need to master the complexities of Oracle Analytics Cloud (OAC).

The feature’s ability to generate instant visual insights from natural language inputs is particularly valuable. This means that once your dataset is indexed and ready, you can quickly derive meaningful insights, spot trends, or explore relationships in the data without having to dive into manual visualisation processes. Auto Insights can also uncover patterns you may not have initially considered, adding further value by highlighting hidden connections.

Subject Areas
This version of the AI Assistant does not currently have the capability to support subject areas. However, this feature is part of the planned future enhancements and is expected to be integrated into the Assistant in due course. Once implemented, it will allow users to engage with the Assistant more effectively, offering deeper insights and tailored assistance based on specific subject areas. 

Update - 19th September 2024

After some conversations with the Product Management team , I have gathered some more information about the AI Assistant pertaining to the rollout process and some details about security and data privacy which you can find below:.

Oracle’s objective is to make the AI Assistant available at no additional cost across all OCPU shapes. However, its release will be managed through a phased, controlled rollout that will span several Oracle Analytics Cloud updates. This approach ensures optimal performance and stability while addressing key concerns around privacy and security.

The Large Language Model (LLM) powering the Assistant is highly resource-intensive and Oracle-managed. Due to its proprietary nature and status as protected intellectual property, specific details regarding the LLM are limited. However, it is important to note that each Oracle Analytics Cloud instance is provisioned with its own private, Oracle-managed LLM.

To further safeguard data privacy, all interactions between the LLM and each individual Oracle Analytics Cloud instance remain entirely within the customer’s OCI environment. At no point do these interactions leave the customer’s infrastructure, ensuring complete data security and confidentiality.

Generated with AI ∙ 16 July 2024 at 11:02 PM

The Oracle Analytics Cloud July 2024 update is on its way out to the general public as this blog is being written. We will focus our attention on one of the new features that significantly improves the benefits of dataset in the Oracle Analytics data visualisation tool.

You can watch this YouTube playlist below to find out details of some of the main features in the July 2024 update. For more information on the update, click here to get a comprehensive list of the features that are included in the July 2024 update.

In this blog, we'll focus on the data caching feature improvements which are part of the July 2024 update. Caching the data in datasets provides a number of benefits which include the potential for improved query performance. Being able to apply changes made to the source data has been possible previously by doing a full reload of the dataset. This can be a long drawn out process if the underlying data source has a large volume of data.

In the July 2024 Oracle Analytics Cloud update, it is now possible to incrementally update datasets. There is now the capability to only insert new rows and updates to apply any changes to existing data. which will be much better than a full data reload. This improves the data load time as well as the execution of queries against the dataset.

These performance improvements that caching provides come with a downside. If you require real time data, caching is not the option for you. If intraday data changes aren't critical to your business area that you are analysing, then data caching will work for you.

Business Benefits

Incrementally loading data into a dataset in Oracle Analytics Cloud (OAC) offers several business benefits:

Reduced System Load and Improved Performance:
 By only loading new or changed records, the system avoids reprocessing the entire dataset. This reduces the overall load on the system, leading to better performance and quicker data refresh times.

Minimised Downtime:
Incremental loading can be scheduled during off-peak hours or more frequently, ensuring that the system is always up-to-date without significant downtime or disruption to users.

Faster Data Refresh:
Since only new or updated data is processed, the time required to refresh datasets is significantly reduced. This allows for more timely and accurate data availability for business analysis and decision-making.

Scalability:
Businesses can scale their data operations more effectively with incremental loading, handling larger datasets without the need for massive resource increases. This supports business growth and the integration of more data sources.

Enhanced Data Accuracy and Relevance:
 With more frequent and efficient data updates, businesses can ensure that their analytics are based on the most current data. This leads to more accurate insights and better-informed decision-making.

Improved User Experience:
End users experience faster query responses and up-to-date data, which enhances their ability to perform timely and effective analysis. This leads to higher user satisfaction and greater adoption of the analytics platform.

Implementing incremental data loading in Oracle Analytics Cloud supports these benefits, enabling businesses to maintain efficient, scalable, and cost-effective data management practices. Data caching is not available to all data sources. You can get a full list of supported data sources from here.

There are 2 methods that can be used to set the incremental dataset cache refresh; this can be done either in a visualisation or directly within the dataset itself.

Configuration in a Visualisation

​The image below shows you how to set this up in a visualisation.

Change the Data Access from Live to Automatic Caching and then change the Cache Reload Type to Load New and Updated Data. 

Configuration in ​a Dataset

The image below shows how the incremental caching is set up in a dataset.

As described above, you can set up a variety of data sources to refresh incrementally and these can be run manually as above. If there is a requirement for the dataset to be regularly refreshed then a schedule can be set up to enable this. Below, you can see how to set up a schedule that can be used to automate the cache refresh of the dataset. You can also view the details of the schedule and detail of the schedule executions. 

Incremental data loading of datasets in Oracle Analytics Cloud offers significant business benefits by enhancing system performance and efficiency. By only loading new or updated records, it reduces system load, minimizes downtime, and accelerates data refresh times. This approach optimises resource utilisation, providing cost savings and enabling scalability for larger datasets. Additionally, it ensures data accuracy and relevance, improving the user experience with faster query responses and up-to-date information.

​Overall, incremental loading supports efficient, scalable, and cost-effective data management practices, crucial for informed decision-making and business growth.

Oracle recently made the latest version of their database: Oracle Database 23ai generally available, the next generation of its database incorporates a number of AI capabilities. There are over 300 new features in this new release and we'll be looking at some of these features from an Analytics perspective, empowering you to unlock deeper insights from your data.

Some of the highlights of the Oracle Database 23ai release are discussed in this video below. 

Unified Data Access: By storing and managing diverse data types within a single database system, a converged database eliminates the need for separate data silos and specialised databases for different workloads. This unified data access makes it easier to query, analyse, and generate insights across multiple data types without the overhead of complex data integration or movement between different systems.

Consistent Data Governance: With all corporate data workloads stored in a converged database, it becomes easier to implement consistent data governance policies, security controls, and access management across the entire data landscape. This ensures data integrity, compliance, and proper access controls when generating insights from various workloads.

Cross-Workload Analysis: A converged database allows you to perform cross-workload analysis by combining data from different sources and types. For example, you could analyse spatial data in the context of relational data, or enrich graph data with information from blockchain workloads. This cross-pollination of data can lead to more comprehensive and valuable insights.

Unified Query Processing: By having a single query processing engine that can handle diverse data types, a converged database can optimise query execution and leverage advanced techniques like vectorisation, parallelisation, and indexing across different workloads. This can lead to faster and more efficient insight generation, especially for complex queries spanning multiple data types.

Simplified Data Pipelines: Instead of maintaining separate data pipelines and ETL processes for each workload, a converged database allows you to streamline data ingestion, transformation, and analysis workflows, reducing complexity and potential sources of error.

Improved Collaboration: With all corporate data workloads residing in a single converged database, it becomes easier for different teams and departments within an organisation to collaborate, share data, and generate insights collectively. This can foster better cross-functional analysis and decision-making.

Future-Proofing: As new data types and workloads emerge, a converged database architecture can more easily adapt and incorporate them, future-proofing your data infrastructure and reducing the need for frequent migrations or replacements of specialised databases.

​AI Vector Search is another feature that can improve insights derived from your corporate data as it provides the functionality to access data in all the formats allowing foundation models like OpenAI's ChatGPT, Microsoft's Copilot, Anthropic's Claude to name a few to provide more accurate responses with access to all data in the Oracle Database 23ai.

AI Vector Search
​Another noteworthy feature introduced in Oracle Database 23ai is AI Vector Search. This functionality has the potential to significantly improve how users gain insights from their data. Unlike traditional keyword searches, AI Vector Search goes beyond simply matching terms. Instead, it focuses on understanding the underlying meaning and relationships within your data. This is achieved by converting text, images, and even relational data into mathematical representations that capture their essence. By comparing these representations, the database can identify information that aligns most closely with your query, regardless of its format or specific wording. This more semantic approach to searching unlocks new possibilities for data exploration within Oracle Analytics Cloud. Users can uncover previously hidden connections and patterns within their data, leading to a richer understanding and the ability to make more informed decisions.

AI Vector Search is considered a significant benefit of a converged database when it comes to improving generated insights across a wide range of corporate workloads, including relational, spatial, graph, and blockchain workloads. Here's how AI Vector Search can enhance insight generation in a converged database environment:

Unified Data Representation: By representing diverse data types as vectors, AI Vector Search allows for a consistent and unified representation of data across different workloads. This enables the application of machine learning and AI techniques consistently, regardless of the underlying data structure or format.

Semantic Search: AI Vector Search leverages the power of vector representations to capture the semantic relationships between data elements. This enables more meaningful and context-aware searches, leading to better insights by retrieving relevant information based on conceptual similarity rather than just exact matches.

Cross-Workload Similarity Analysis: With vector representations, AI Vector Search can identify similarities and patterns across different workloads, even if the data types are distinct. For example, it could uncover relationships between spatial data and graph data, or find connections between blockchain transactions and relational data, enabling novel insights and discoveries.

Improved Query Performance: AI Vector Search typically uses specialised indexing techniques optimised for vector data, enabling faster and more efficient querying across large datasets spanning multiple workloads. This can significantly accelerate the process of generating insights, especially for complex analytical queries.

Scalability: AI Vector Search can scale more efficiently than traditional search methods, as vector representations are typically more compact and can leverage techniques like dimensionality reduction. This scalability is crucial when dealing with the ever-increasing volumes of data across diverse corporate workloads.

Integration with AI/ML Models: The vector representations used in AI Vector Search are compatible with many AI and machine learning models, enabling seamless integration of these advanced techniques for enhancing insight generation. For example, natural language processing models could be used to query and analyse text data across different workloads.

I'd strongly recommend spending a bit of time taking this LiveLabs workshop to get an understanding of the vector data type basic principles. 

Here's a breakdown of the steps to configure this setup:
​
1. Setting Up SharePoint Permissions:

• Ensure Oracle Database 23ai has read access to the relevant document libraries within your SharePoint site. This might involve configuring crawl permissions for an appropriate service account used by Oracle Database 23ai.

2. Oracle Database 23ai Configuration:

• Within Oracle Database 23ai, define an external data source pointing to your SharePoint site. This typically involves specifying connection details and authentication methods.
• Metadata Definition: Define the metadata fields you want to store within Oracle Database 23ai. This could include properties like document titles, creation dates, authors, or even custom tags associated with the documents in SharePoint.
• Vector Indexing: Create vector indexes for the chosen metadata fields. This allows Oracle Database 23ai to perform semantic searches based on the meaning conveyed through the metadata.​

3. Oracle Analytics Cloud Configuration:

• Establish a secure connection between Oracle Database 23ai and Oracle Analytics Cloud.
• Define the data source within Oracle Analytics Cloud, pointing it to the relevant database instance and tables in Oracle Database 23ai containing the SharePoint metadata.

4. Utilising NLP in Oracle Analytics Cloud:

• Design dashboards and visualisations within Oracle Analytics Cloud that allow users to interact with the SharePoint data through NLP queries.
• Users can then ask questions related to the documents stored in SharePoint based on the available metadata (e.g., "find all documents about marketing campaigns created last quarter").

Benefits:

• Centralised Search: Users can search for relevant documents across your SharePoint site using a single platform (Oracle Analytics Cloud) with natural language queries.
• Improved Discoverability: Leverage AI Vector Search to find documents based on meaning within the metadata, not just filenames or basic keywords.
• Flexibility: You can keep your documents within the familiar SharePoint environment while enabling a more powerful search experience through Oracle Database 23ai.

Important Considerations:

• This is a general guideline, and the specific configuration steps might vary depending on your SharePoint version and Oracle Database 23ai setup. Refer to the official documentation from Microsoft (SharePoint) and Oracle for detailed instructions.
• Security measures are crucial to ensure proper access control to both the SharePoint documents and the metadata stored within Oracle Database 23ai.
• Performance considerations exist when dealing with vast amounts of documents and metadata. Optimising data extraction and indexing processes might be necessary.

By following these steps and considerations, you can leverage Oracle Database 23ai's AI Vector Search capabilities to unlock a more intuitive and semantic search experience for your SharePoint documents within Oracle Analytics Cloud.

By leveraging the AI Vector Search feature, the company can gain deeper and more nuanced insights from their data, leading to better-informed business decisions and a more responsive approach to customer feedback

However, it's important to note that realising the full benefits of a converged database for generating insights across diverse workloads may require careful data modelling, performance optimisations, and leveraging the advanced analytical capabilities provided by the database system, such as machine learning, graph analytics, and spatial analysis functions.​​​​​​​​​​​​​​​​

Conclusion
​In conclusion, Oracle Database 23ai with its converged database architecture and AI Vector Search has the potential to unlock a new level of data exploration within Oracle Analytics Cloud. By storing all your corporate data, structured and unstructured, in a single platform, you can leverage the power of AI to search for insights across different data types using natural language. This not only simplifies data exploration and data integration for users but also empowers them to uncover hidden connections and patterns that might be missed with traditional keyword searches. As a result, Oracle Database 23ai paves the way for a more intuitive and insightful data analysis experience within Oracle Analytics Cloud, ultimately leading to better data-driven decision making.

When you create a dataset in Oracle Analytics, the system performs column-level profiling to generate semantic recommendations for data enrichment. Additionally, when creating workbooks, you can enhance visualisations by including knowledge enrichments from the Data Panel. These recommendations are based on automatic detection of specific semantic types during the profiling process. 
​
Semantic types include geographic locations identified by city names, recognisable patterns (such as credit card numbers, email addresses, and social security numbers), dates, and recurring patterns.

In the share price dataset below which has a date column selected, the panel on the right hand side provides semantic recommendations related to the date column which can be added to the dataset to provide additional classifications in addition to the original date from the data set like the month, quarter and year. 

This is very helpful in adding new ways to look at data but this is limited to the system built-in semantic recommendations  that come with the Oracle Analytics platform. Most companies have domain specific knowledge which would be useful in this context to provide enrichment possibilities to users. 

In the sample file below which contains geographic information about airport codes, this can be used to give additional geographic context to datasets that only contain the 3-digit airport code. 

To demonstrate this feature, we will be adding some custom knowledge that can be used to enrich 3-digit airport codes with additional information about the airport name, country and continent. 

As an Administrator, click the Reference Knowledge button in the Oracle Analytics administration console. 

Select the file (either an XSLX or CSV format) with the custom knowledge to upload.  I have noticed that the first column in the data to be uploaded to custom knowledge should be the unique key used to identify each record uniquely.  In the case above, the 3-digit airport code is unique and should be the first column in the data. 

​Review the content in the window below and click OK if all looks good. 

Now that the custom knowledge has been uploaded to Oracle Analytics, it is now available to be used in workbooks that have the airport code in their datasets. This workbook has a list of London airports and their approximate passenger numbers form 2014 - 2023.  We can make use of the custom knowledge to enrich this workbook.

Navigate to the Data tab and you should now see recommendations for the Airport column with options as seen in the screenshot below. 

The enrichments can be seen here and will now be available to use within the workbook. 

Alternatively, the enrichments can also be seen directly in the workbook data panel enabling non-administrator users to have access to these enrichments. 

What would improve this feature would be having an option to derive the custom knowledge from a database source for the purposes of data governance. A refresh schedule would be a welcome addition to the feature as well ensuring that the custom knowledge is sourced directly from a database but also routinely refreshed to ensure that the custom knowledge is kept up to date. ​

This functionality serves as a valuable tool that can empower users to achieve a deeper understanding of their data and extracting insights that have greater relevance to the end user.