Agentic coding

 

We have all experienced the "maintenance wall"—that point where a hobby coding project becomes so complex that adding a single feature feels like a massive effort and a gamble. For years, my side projects and hobbies lived in this state of stagnation: functional, but fragile and difficult to extend. Everything changed when I integrated Claude Code  from Anthropic into my development workflows.

By treating Claude Code as an agentic code writing partner, a co-programmer - remember XP programming - and rather than a simple chatbot, I moved beyond just "writing code" and began deploying fast, really fast. Before getting started, a quick reminder: these tools can be token-intensive. You may end up spending a lot of tokens (new money). Always keep your repository structure in mind, optimise and whenever possible, feed your repository context to Claude Code to keep interactions efficient and economical. Use CLAUDE.MD to restrict scope and .claudeignore to ignore files. 

Beyond Syntax: Learning Why

The most significant shift in my workflow was transitioning from "fixing bugs" to "improving architectural understanding." I stopped asking how to write a line of code and started asking why my current implementation was failing to scale and what would the improvements be.

By prompting Claude Code to analyze my existing modules and explain it to me, I uncovered limitations I hadn't even realised were there. I learned what I was doing wrong and what I was omitting which was important. It pointed out anti-patterns like tightly coupled state management and hidden resource bottlenecks. It highlighted things that were security flaws, bad session management, inefficient and bloaty code. Patiently it explained to me things, recommended alternatives, implemented them on my code and help me deploy them. 

Expanding the Horizon

Once I understood the structural limitations of my projects, Claude Code became the engine that allowed me to expand their scope significantly:

• UI/UX Enhancement: I leveraged the tool to audit my frontend components. By describing my goals for user intuition and accessibility, I received actionable critiques on layout and CSS architecture that transformed my interfaces from "functional" to "polished."

• Security by Design: Security is often the first casualty in hobby development. I used Claude Code to scan my deployment scripts and configuration files for exposed secrets and insecure defaults. It taught me to bake security into my deployment pipeline as a baseline requirement, not an afterthought.

• The Refactor Loop: Perhaps the most liberating benefit I saw was the ability to refactor non stop and fearlessly. By having an agent that understands the relationships across my entire repository, I could rewrite core logic, simplify complex functions, and modularise monolithic loooong blocks with the confidence that Claude had analyzed the ripple effects.

• Velocity Through Testing: With the Model Context Protocol (MCP), I connected my codebase to my testing environment. This bridge allowed Claude to write code that was verified by real-time test execution. Writing code that is "correct by construction" with a safety net of tests drastically increased my development velocity.

The real game-changer was shifting from treating Claude as a chatbot to treating it as an autonomous agent, a guru co-programmer like in the XP and pair-programming days.  By leveraging CLAUDE.md for persistent architecture rules and creating custom Skills for my repetitive workflows, I’ve essentially given my projects an 'institutional memory' context. I realised I can even start utilising Agent Teams (the hiring of programmers), where I can orchestrate sub-agents to handle security audits and testing in parallel, while I focus on the high-level design. But, I haven't done that yet, I am thinking what would that do to my mere hand counted tokens in my simple subscription 😊

Now the exciting part, using Model Context Protocol (MCP) - the eyes, hands and legs of AI, the tools - I was able to connect claude-code to my !test database, get claude query the tables, turning my terminal into a command center with well informed insights, that doesn't just write code, but actively manages, tests and executes it. WARNING! Be careful of this, know what you are doing! To be honest, I don't let it near production or to deploy my code yet, without me checking it,  I do it myself with vscode. 

 Final Thoughts

Using Agentic coding with AI didn't make me a passive coder; it made me a more rigorous one. I spend less time wrestling with syntax, CSS and statics and more time exploring advanced architectural concepts that previously felt out of reach. If you have a project sitting on the shelf, I encourage you to bring it into the terminal. Just remember to be specific, keep an eye on your token economy! 

Let agentic coding infuse your projects with the velocity to innovate rapidly and the structural strength to evolve fast.

 

Why trustworthy AI isn’t optional - it’s the foundation

In the rapidly evolving landscape of generative AI, data security and data privacy are not just compliance requirement - they are the bedrock of trust and innovation. 

After all who would want to waste time with a dodgy and fake AI?😑

While data security focuses on protecting AI models and datasets from breaches, tampering, or unauthorised access (through measures like encryption, access controls, and secure APIs), data privacy ensures that the data powering these models is collected, processed, and used ethically and legally. This distinction becomes particularly critical in grounding techniques, where AI models are anchored to external knowledge bases, APIs, or real-time data sources. The more an AI model is trained on accurate, well-structured data, the fewer inaccuracies and hallucinations it produces.

What is grounding?

In simple terms, grounding connects AI models to external sources of truth - such as sensors, databases, or APIs - to ensure responses are accurate, context-aware, and reliable. Without grounding, AI risks hallucinations or relying on outdated or biased data.

For example, imagine grounding an AI customer support chatbot in a SQL database of product manuals and FAQs. The chatbot queries the database in real-time to provide accurate, up-to-date answers—rather than inventing responses. However, if the database connection isn’t secured with encryption or the data isn’t anonymized, the grounded AI system could expose sensitive customer information or fall victim to prompt injection attacks.

And without robust security, grounded AI systems risk exposing sensitive data or being manipulated through techniques like prompt injection or data poisoning. Without privacy safeguards, they may inadvertently violate regulations like GDPR or CCPA by misusing or retaining personal data embedded in their knowledge sources.

For data architects, data engineers and AI engineers, grounding introduces unique challenges. When an AI model queries external databases, APIs, or data lakes, every interaction must be both secured (e.g., using TLS 1.3, OAuth 2.0, or zero-trust architectures) and privacy-preserving (e.g., via differential privacy-federated learning, or data anonymization). For example, if you’re grounding a large language model (LLM) in a relational database or Delta Lake, you MUST ensure that:

Security: The connection is encrypted, access is role-based (e.g., RBAC in Snowflake or Azure Synapse), and queries are logged for audits.

Privacy: The underlying data is scrambled, anonymised or pseudonymized, and the model only retrieves data it’s authorised to use - aligning with the principle of least privilege. Tools like Python’s faker or SQL’s dynamic data masking can help strip PII from responses, your grounding frameworks MUST enforce strict data access policies.

The Future: Trustworthy Grounding in AI

The future of AI hinges on trustworthy grounding - where models don’t just perform well but also respect data sovereignty and user consent. As you design AI systems that interact with databases, data lakes, or Lakehouses, prioritise :

Privacy-by-Design: Embed consent checks into API calls, database connections and ensure data minimisation, authorised data only. 

Security-by-Default: Encrypt even vectors-embeddings in your vector databases and all related grounding data assets and enforce strict access controls.

If you’re experimenting with grounding, start by auditing your data sources: 

• Are they secure? 
• Are they privacy-compliant? 

The answers will define whether your use of AI with your database is not just smart, but also trustworthy.

Exploring C4 Models with Structurizr DSL, VSCode, and Diagramming Tools

Introduction

As a Data Architect, creating clear and effective diagrams is crucial for communicating and documenting software and data architectures. The C4 model, with its focus on abstraction-first design—a principle I firmly believe is the backbone of software engineering—immediately caught my interest. To explore this further, I recently began experimenting with C4 modeling using Structurizr DSL, (DSL=Domain Specific Language) VSCode, and popular diagramming tools like PlantUML and Mermaid. I used Cairo and Graphviz in the past but these newer libraries require less tinkering. Here’s a look at my journey and the insights I gained along the way while trying the diagram as code approach.

Why C4 Models?

The C4 model is a powerful way to describe software systems at four distinct levels of abstraction—Context, Containers, Components, and Code—often referred to as the "4 Cs." Its simplicity, scalability, and developer-friendly approach make it a perfect fit for both new (greenfield) and existing (brownfield) projects.

Since I prefer to avoid cloud-based tools for richer experience and control, initially I set up a local environment using VSCode and Docker on my trusty old but fast Ubuntu laptop. This way, I can create clear, maintainable diagrams while keeping everything offline and efficient. Looking at it again, I decided that even Docker is an overkill. I decided Vscode is enough to code and diagram.

My Setup

I took a quick look at the Structurizr DSL Python wrapper, but I also skipped it—I wanted to dive straight into the native DSL syntax and see my diagrams render with minimal overhead. After all, treating diagrams as code means I can version-control them like any other project, keeping everything clean and reproducible.

While I could have spun up Structurizr Lite in a Docker container (because who doesn’t love local, self-hosted solutions?), I went lighter—just VSCode extensions to get the job done. My philosophy? Minimum viable effort for maximum results. No unnecessary layers, no cloud dependencies, just code and diagrams, the way it should be. 

They integrate seamlessly with wiki platforms (like Confluence, Notion, or GitLab/GitHub Wikis) and Git repositories, allowing you to embed dynamic, version-controlled diagrams directly in your documentation.

Tools in Action

• Structurizr DSL: Writing diagrams as code in DSL in vscode and for better previews run their server on localhost
• VSCode: With extensions for PlantUML and Mermaid, I could preview diagrams instantly in vscode.
• PlantUML & Mermaid: Both tools integrated seamlessly with VSCode via extensions, though I found Mermaid’s syntax more intuitive for quick sketches and wiki integration. Mermaid has its own markup.

Outcomes

I successfully created Context, Container, and Component diagrams for a sample imaginary project. The ability to generate diagrams locally ensured full control and flexibility, no SaaS. Here are two examples of what I built:

Figure 1: Output from Structurizr server running on localhost:8080 in docker with code on the left generating the C4 model diagram on the right

Figure 2: Output from Mermaid vscode extension showing Mermaid code on the left generating the diagram on the right

Final Thoughts

I find the C4 model and tools like PlantUML and Mermaid are a game-changer for architecture documentation—it shifts the process from static, manual diagrams to code-driven, version-controlled clarity. By leveraging Structurizr DSL in VSCode and pairing it with Mermaid/PlantUML, I’ve crafted a workflow that’s both flexible and precise, giving me full control over how my systems are visualized.

There’s something deeply satisfying about coding your diagrams—no more wrestling with drag-and-drop tools or misaligned Bézier curves. Just clean, maintainable DSL and instant visual feedback. I’m officially done with joining rectangles by hand; from now on, it’s code all the way.

Building a Real-Time Weather App with Streamlit and Open-Meteo

see app here: https://data-exchange.streamlit.app/

I recently embarked on a project to build a real-time weather application, and I wanted to share my experience using Streamlit and Open-Meteo. The goal was to create a dynamic web app that provides users with up-to-date weather information and 10 day weather forecasts, all while leveraging the convenience of cloud-based development.

Streamlit: Rapid Web App Development in the Browser

One of the most compelling aspects of Streamlit is its ability to facilitate rapid web application development directly within the browser. For this project, I utilized GitHub Codespaces, which provided a seamless development environment. This eliminated the need for complex local setups and allowed me to focus solely on coding.

Key Advantages of Using GitHub Codespaces:

• Browser-Based Workflow: All development activities were performed within a web browser, streamlining the process.
• Dependency Management: Installing necessary Python packages was straightforward using pip install.
• Version Control: Integrating with Git enabled efficient version control with git commit and push commands.

Data Acquisition with Open-Meteo

To obtain accurate and current weather data, I employed the Open-Meteo API. This API offers a comprehensive set of weather parameters, allowing for detailed data visualization.

Visualizing Weather Data with Streamlit's Graph Capabilities

Streamlit's built-in graph visualization tools proved to be highly effective. Creating dynamic charts to represent weather data was quick and efficient. The clarity and responsiveness of these visualizations significantly enhanced the user experience.

Technical Implementation:

The application was developed using Python, leveraging Streamlit for the front-end and the requests library to interact with the Open-Meteo API. The workflow involved:

1. Fetching weather data from the Open-Meteo API.
2. Processing the data to extract relevant information.
3. Utilizing Streamlit's charting functions to create graphical representations.
4. Deploying the application via the streamlit sharing platform.

Observations:

• Streamlit's simplicity and ease of use allowed for rapid prototyping and development.
• GitHub Codespaces provided a consistent and reliable development environment.
• Open-Meteo API provided accurate data.
• Streamlit sharing made deployment very easy.

Conclusion:

This project demonstrated the power of Streamlit and Open-Meteo for building data-driven web applications. The ability to develop entirely within a browser, combined with powerful visualisation tools, made the development process efficient and enjoyable.

You can view the final app here: https://data-exchange.streamlit.app/

GitHub Codespaces: A Fast-Track to Development with Minimal Setup

Do you like coding but you hate the scaffolding and prep-work?

As developer, I often spend a considerable amount of time setting up development environments and the project scaffolding before I even write a single line of code. Configuring dependencies, installing tools, and making sure everything runs smoothly across different machines can be tedious. IF you find this prep work time consuming and constraining then...

Enter GitHub Codespaces 

GitHub Codespaces is  cloud-based development environment that allows you to start coding instantly without the hassle of setting up a local machine on your browser!

Whether you’re working on an open-source project, collaborating with a team, or quickly prototyping an idea, Codespaces provides a streamlined workflow with minimal scaffolding.

Why GitHub Codespaces?

1. Instant Development Environments
   With a few clicks, you get a fully configured development environment in the cloud. No need to install dependencies manually—just launch a Codespace, and it’s ready to go.

2. Pre-configured for Your Project
   Codespaces can use Dev Containers (.devcontainer.json) to define dependencies, extensions, and runtime settings. This means every team member gets an identical setup, reducing "works on my machine" issues.

3. Seamless GitHub Integration
   Since Codespaces runs directly on GitHub, pushing, pulling, and collaborating on repositories is effortless. No need to clone and configure repositories locally.

4. Access from Anywhere
   You can code from a browser, VSCode desktop, or even an iPad, making it an excellent option for developers who switch devices frequently.

5. Powerful Compute Resources
   Codespaces provides scalable cloud infrastructure, so even resource-intensive projects can run smoothly without overloading your local machine.

A Real-World Example

Imagine you’re starting a new Streamlit project on their community. Normally, you’d:

• Install Streamlit and other packages
• Set up a virtual environment
• Configure dependencies
• Ensure all team members have the same setup

With GitHub Codespaces, you can define everything in a requirements.txt and .devcontainer.json file and launch your environment in seconds. No more worrying about mismatched Python versions or missing dependencies—just open a browser and start coding. 

See below how I obtained this coding environment to built a Weather Streamlit app quickly and for FREE using the Streamlit community Cloud 

All in one browser page using GitHub, browser edition of VScode and access to a free machine on Streamlit Community Cloud with GitHub Codespace for development.

To see the above app visit https://click-weather.streamlit.app/

Final Thoughts

GitHub Codespaces is a game-changer for modern development I think. It eliminates the friction of setting up environments, making collaboration effortless and speeding up development cycles. If you haven’t tried it yet, spin up a Codespace for your next project—you might never go back to traditional setups on your laptop anymore. 

There is another tool I want to look at which does all the scaffolding automatically with AI. Is the IDE called 'Windsurf' from Codeium, but that's another blog post.

Understanding the Contextual Awareness of data

    Image attribution: Thomas Nordwest, CC BY-SA 4.0, via Wikimedia Commons

What sources does this data come from?

How was this data collected in the first place?

Do you clearly understand the structure and format of your data?

What specific business objectives does this data help you achieve?

Do you know that there are relationships between this dataset and others, if you do which?

What limitations or biases there may exist in this data?

How frequently is this data updated or refreshed, is it stale?

What context is important to consider when trying to understand this data?

What questions are you trying to answer with this data?

 
The above questions can help us achieve data zen, or the contextual awareness we all seek.

Contextual Awareness of data refers to the ability to understand and interpret data within its relevant situation, including the source, structure, and intended use. It is the 'world of interest' it tries to describe. It goes beyond merely collecting data; it encompasses a comprehensive understanding of what the data represents, where it originates, and how it relates to other datasets. This awareness is crucial for organisations to derive meaningful insights, as data without context can lead to misinterpretation and ineffective decision-making.

The importance of contextual awareness cannot be overstated. Without a clear understanding of the data’s context, organisations risk making decisions based on incomplete or misleading information. For instance, data collected from disparate sources may appear accurate in isolation but may contradict one another when analysed together. By fostering contextual awareness, organisations can ensure they ask the right questions and identify relevant insights, ultimately driving strategic initiatives and operational efficiency.

Furthermore, contextual awareness enhances data quality assessment and improves the overall data governance framework. When organisations know the context of their data, when they know the semantics or the meaning of their data, they can better assess its reliability and relevance. This clarity allows for more informed decision-making processes, ensuring that data insights are actionable and aligned with business objectives. In today's data-driven landscape, cultivating contextual awareness is not just useful—it is essential for achieving the competitive edge and fostering a data-driven culture within the organisations.

Several tools like conceptual data models, data profiling, talking to colleagues, data catalogs and ontologies can all help you in the journey of understanding your data.

So ask loads of questions the next time you see that data set!

Unraveling Data Architecture: Data Fabric vs. Data Mesh

In this first post of this year, I would like to talk about modern data architectures and specifically about two prominent models, Data Fabric and Data Mesh. Both are potential solutions for organisations working with complex data and database systems.

While both try to make our data lives better and try to bring us the abstraction we need when working with data, they are different in their approach.

But when would be best to use one over the other? Let's try to understand that.

Definitions Data Fabric and Data Mesh

Some definitions first. I come up with these after reading up on the internet and various books on the subject.

Data Fabric is a centralised data architecture focusing on creating a unified data environment. In it's environment data fabric integrates different data sources and provides seamless data access, governance and management and often it does this via tools and automation. Keywords to remember from the Data Fabric definition are centralised, unified, seamless data access.

Data Mesh, is a paradigm shift, it is a completely different way of doing things. Is based on domains, most likely inspired by the Domain Driven Design (DDD) example, is capturing data products in domains by decentralising data ownership (autonomy) and access. That is, the domains are the owners of the data and are responsible themselves for creating and maintaining their own data products. The responsibility is distributed. Key words to take away from Data Mesh are decentralised, domain ownership, autonomy and data products.

Criteria for Choosing between Data Fabric and Data Mesh

Data Fabric might be preferred when there is a need for centralised governance and control over the data entities. When a unified view is needed across all database systems and data sources in the organisation or departments. Transactional database workloads are very suitable for this type of data architecture where consistency and integrity in their data operations is paramount.

Data Mesh can be more suitable for organisational cultures or departments where scalability and agility is a priority. Because of its domain-driven design, Data Mesh might be a better fit for organisations or departments that are decentralised, innovative, and require their business units to swiftly and independently decide how to handle their data. Analytical workloads and other big data workloads may be more suitable to Data Mesh data architectures.

Ultimately, the decision-making process between these data architectures hinges on the load of data processing and the alignment of diverse data sources. There's no universal solution applicable to all scenarios or one size fits all. Organisations and departments in organisations operate within unique cultural and environmental contexts, often necessitating thorough research, proof of concept, and pattern evaluation to identify the optimal architectural fit.

Remember, in the realm of data architecture, the data workload reigns supreme - it dictates the design.

Using vector databases for context in AI

Image generated by AI using OpenAI's DALL·E

In the realm of Artificial Intelligence (AI), understanding and retaining context stands as a pivotal factor for decision-making and enhanced comprehension. Vector databases, are the foundational pillars in encapsulating your own data to be used in conjunction with AI and LLMs. Vector databases are empowering these systems to absorb and retain intricate contextual information.

Understanding Vector Databases

Vector databases are specialised data storage systems engineered to efficiently manage and retrieve vectorised data - also known as embeddings. These databases store information in a vector format, where each data entity is represented as a multidimensional numerical vector, encapsulating various attributes and relationships, thus fostering the preservation of rich context. That is text, video or audio is translated into numbers with many attributes in the multidimensional space. Then mathematics are used to calculate the proximity between these numbers. Loosely speaking that is what a neural network in an LLM does, it computes proximity (similarity) between the vectors. A bit like how our brains do pattern recognition. The vector database is the database where the vectors are stored. Without a vector databases under architectures like RAG, it is impossible to bring own data or context into an LLM app in an AI model as all it will know will be only what it is trained on from the public internet. Vector databases enable you to bring your own data to AI.

Examples of Vector Databases

Several platforms offer vector databases, such as Pinecone, Faiss by Facebook, Annoy, Milvus, and Elasticsearch with dense vector support. These databases cater to diverse use cases, offering functionalities tailored to handle vast amounts of vectorised information, be it images, text, audio, or other complex data types.

Importance in AI Context

Within the AI landscape, vector databases play a pivotal role in serving specific data and context for AI models. Particularly, in the Retrieval-Augmented Generation (RAG) architecture, where retrieval of relevant information is an essential part of content generation, vector databases act as repositories, storing precomputed embeddings from your own private data. These embeddings encode the semantic and contextual essence of your data, facilitating efficient retrieval in your AI apps and Bots. Bringing vector databases to your AI apps or chatbots will bring your own data to your AI apps, agents and chatbots will speak your data!

Advantages for Organisations and AI Applications

Organisations can harness the prowess of vector databases within Retrieval-Augmented Generation (RAG) architectures to elevate their AI applications and enable them to use organisational specific data:

1. Enhanced Contextual Understanding: By leveraging vector databases, AI models grasp nuanced contextual information, enabling more informed decision-making and more precise content generation based on specific and private organisational context.

2. Improved Efficiency in Information Retrieval: Vector databases expedite the retrieval of pertinent information by enabling similarity searches based on vector representations, augmenting the speed and accuracy of AI applications.

3. Scalability and Flexibility: These databases offer scalability and flexibility, accommodating diverse data types and expanding corpora, essential for the evolving needs of AI-driven applications.

4. Optimised Resource Utilisation: Vector databases streamline resource utilisation by efficiently storing and retrieving vectorised data, thus optimising computational resources and infrastructure.

Closing Thoughts

In the AI landscape, where the comprehension of context is paramount, vector databases emerge as linchpins, fortifying AI systems with the capability to retain and comprehend context-rich information. Their integration within Retrieval-Augmented Generation (RAG)  architectures not only elevates AI applications but also empowers organisations to glean profound insights, fostering a new era of context-driven AI innovation from data.

In essence, the power vested in vector databases will reshape the trajectory of AI, propelling it toward unparalleled contextualisation and intelligent decision-making based on in house and organisations own data.

But the enigma persists: What precisely will be the data fuelling the AI model?

Vscode container development

If you're a software developer, you know how important it is to have a development environment that is flexible, efficient, and easy to use. PyCharm is a popular IDE (Integrated Development Environment) for Python developers, but there are other options out there that may suit your needs better. One such option is Visual Studio Code, or VS Code for short.

After using PyCharm for a while, I decided to give VS Code a try, and I was pleasantly surprised by one of its features: the remote container development extension. This extension allows you to develop your code in containers, with no footprint on your local machine at all. This means that you can have a truly ephemeral solution, enabling abstraction to the maximum.

So, how does it work? First, you need to create two files: a Dockerfile and a devcontainer.json file. These files should be located in a hidden .devcontainer folder at the root location of any of your GitHub projects.

The Dockerfile is used to build the container image that will be used for development. Here's a sample Dockerfile that installs Python3, sudo, and SQLite3:

FROM ubuntu:20.04

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update -y

RUN apt-get install -y python3

RUN apt-get install -y sudo

RUN apt-get install -y sqlite3

The devcontainer.json file is used to configure the development environment in the container. Here's a sample devcontainer.json file that sets the workspace folder to "/workspaces/alpha", installs the "ms-python.python" extension, and forwards port 8000:

{

    "name": "hammer",

    "build": {

        "context": ".",

        "dockerfile": "./Dockerfile"

    },

    "workspaceFolder": "/workspaces/alpha",

    "extensions": [

        "ms-python.python"

    ],

    "forwardPorts": [

        8000

    ]

}

Once you have these files ready, you can clone your GitHub code down to a Visual Studio Code container volume. Here's how to do it:

1. Start Visual Studio Code
2. Make sure you have the "Remote Development" extension installed and enabled
3. Go to the "Remote Explorer" extension from the button menu
4. Click "Clone Repository in Container Volume" at the bottom left
5. In the Command Palette, choose "Clone a repository from GitHub in a Container Volume" and pick your GitHub repo.

That's it! You are now tracking your code inside a container volume, built by a Dockerfile which is also being tracked on GitHub together with all your environment-specific extensions you require for development.

The VS Code remote container development extension is a powerful tool for developers who need a flexible, efficient, and easy-to-use development environment. By using containers, you can create an ephemeral solution that allows you to abstract away the complexities of development environments and focus on your code. If you're looking for a new IDE or just want to try something different, give VS Code a try with the remote container development extension.

Is Data Hub the new Staging environment?

"A data hub is an architectural pattern that enables the mediation, sharing, and governance of data flowing from points of production in the enterprise to points of consumption in the enterprise” Ted Friedman, datanami.com

Aren't relational databases, data marts, data warehouses and more recently data lakes not enough? Why is there a need to come up with yet another strategy and paradigm for database management?

To begin answering the above questions, I suggest we start looking at the history of data management and figure out how data architecture developed a new architectural pattern like Data Hub. After all, history is important as a famous quote from Martin Luther King Jr. says "We are not makers of history. We are made by history"

Relational architecture

A few decades ago, businesses began using relational databases and data warehouses to store their interests in a consistent and coherent record.  The relational architecture still keeps the clocks ticking with its well understood architectural structures and relational data models. It is a sound and consistent architectural pattern based on mathematical theory which will continue serving data workloads. The relational architecture serves brilliantly the very specific use case of transactional workloads, where the data semantics are defined in advance before any data is stored in any system. If implemented correctly the relational model can become a hub of information that is centralised and easy to query. It is hard to see that the relational architecture could be the reason to cause a paradigm shift into something like a data hub. Most likely is something else. Could it be cloud computing?

When the cloud came, it changed everything. The Cloud brought along an unfathomable proliferation of apps and an incredible amount of raw and unorganised data. With this outlandish amount of disorganised data in the pipes, the suitability of the relational architecture for data storage had to be re-examined and reviewed.  Faced with a data deluge, the relational architecture couldn't scale quickly and couldn't serve the analytical workloads and the needs of the business in a reasonable time. Put simply, there was no time to understand and model data. The sheer weight of the number of unorganised chunks of data coming from the cloud, structured and unstructured, at high speeds, propelled the engineers to look for a new architectural pattern.

Data Lake 

In a data lake, the structured and unstructured data chunks are stored raw and no questions are asked. Data is not organised and is not kept in well-understood data models anymore and it can be stored infinitely and in abundance. Moreover, very conveniently the process of understanding and creating a data model in a data lake is deferred to the future, which is a process known as schema-on-read. We have to admit, the data lake is the new monolith where data is stored only, a mega data dump yard indeed. This new architectural pattern also brought with it the massively parallel (MPP) processing data platforms, tools and disciplines, such as machine learning, which became the standard methods for extracting business insights from the absurd amounts of data found in a data lake. Unfortunately, the unaccounted amounts of unknown data living in a data lake didn't help us understand data better and made the life of engineers even more difficult. Does a data lake have any redundant data or bad data? Are there complex data silos living in a data lake? These are still hard questions to answer and the chaotic data lakes looked like are missing a mediator. 

Data Hub

Could the mediator be a "data hub"? It is an architectural pattern based on the hub and spoke architecture. A data hub, which itself is another database system, integrates and stores critical and important data and metadata for mediation, from diverse and complex transactional and analytical workloads and data sources. Once the data is stored, the data hub becomes the tool to "harmonise" and "enrich" data and then radiate it to the AI, Machine Learning and other enterprise insights and reporting systems, via its spokes.

What's more, while sharing the data in its spokes, the data hub can also help engineers to govern, secure and catalogue the data landscape of the enterprise. The separation of data via mediation from the source and target database systems inside a data hub also offers engineers the flexibility to operate and govern independently of the source and target systems. But this reminds me of something.

If the data hub paradigm is a mediator presented to understand, organise, correct, enrich and put an order in the data chaos of data lake monoliths, doesn't the data hub look similar to the data management practice engineers have been doing for decades and we all know as "Staging"? Is data hub the evolved version of staging?

Conclusion

The most difficult thing in anything you do is to persuade yourself that there is some value in doing it. It is the same when adopting a new architectural pattern as a data management solution. You have to understand where the change is coming from and see the value before you embark on using it. The data upsurge brought by the internet and cloud computing cause changes to made in data architecture and data storage solutions. The data hub is a new architectural pattern in data management introduced to mediate the chaos of fast-flowing data tsunamis around us and we hope it will help us tally everything up.

Related reading

A Peek At the Future of Data Management, Courtesy of Gartner

What is a Data Hub?

Oracle Apex 20.2 REST API data sources and database tables

Oracle Apex 20.2 is out and has a very interesting new feature, REST Data Source Synchronisation

Why is the REST Data Source Synchronization feature interesting? 

Oracle Apex REST Data Source Synchronisation is exciting because it lets you query REST endpoints on the internet on a schedule or on-demand basis and saves the results automatically in database tables.

I think this feature will suit slow-changing data accessible with REST APIs very well. That is, if a REST endpoint data is known to be changing, say few times a day, why should we call the REST endpoint via HTTP every time we wanted to display data on an Apex page? Why would one want to render a page with data over HTTP if that data changes only once a day? Why should we cause network traffic and keep machines busy for data which is not changing often? Or maybe by requirement, you only needed to query a REST endpoint once a day and store it somewhere for data-warehousing.

Wouldn't it be better to store the data in a database table and render it from there every time a page is viewed?

This is exactly what the REST Data Source Synchronisation does. It queries the REST API endpoint and saves the JSON response as data in a database table on a schedule of your choice or on demand. 

For my experiment, I used the Public Free London TfL REST API Endpoint from the TfL API which holds data for TfL transportation disruptions and I configured this endpoint to synchronise with my database table every day at 5am.

I even created the Oracle Apex REST Data source inside the apex.oracle.com platform. I used the TfL API Dev platform provided key to make the call from there to the TfL REST endpoint and I managed to sync it once a day on an Oracle Apex Faceted Search page and some charts.

I was able to do all this with zero coding, just pointing the Oracle Apex REST Data Source I created for the TfL API to a table and scheduling the sync to happen once a day at 5am. 

To see the working app, go to this link: https://apex.oracle.com/pls/apex/databasesystems/r/tfl-dashboard/home

Screenshots of the app below

Oracle Apex Social Sign in

In this post I want to show you how I used the Oracle Apex Social Sign in feature for my Oracle Apex app. Try it by visiting my web app beachmap.info.

Oracle Apex Social Sign in gives you the ability to use oAuth2 to authenticate and sign in users to your Oracle Apex apps using social media like Google, Facebook and others.

Google and Facebook are the prominent authentication methods currently available, others will probably follow. Social sign in is easy to use, you don't need to code, all you have to do is to obtain project credentials from say Google and then pass them to the Oracle Apex framework and put the sign in button to the page which will require authentication and the flow will kick in. I would say at most a 3 step operation. Step by step instructions are available in the blog posts below.


Further reading:

• http://dgielis.blogspot.com/2018/06/facebook-google-and-custom.html
• http://www.grassroots-oracle.com/2019/01/social-sign-in-authentication-scheme.html