Data Integration and Warehousing

Your customer exists five times, with five spellings and five sets of details. This page covers what a data layer does, when a warehouse is worth building, how master data and matching actually work, and what data integration costs an Australian operator.

What is data integration and warehousing?

Data integration gets records out of the systems that hold them, makes them agree, and puts the result somewhere everything else can read. A warehouse is that place: one consistent, historical copy of the business.

Most reporting, forecasting and AI projects stall on the same problem. One customer exists under several names, and no two systems share an ID.

What a data layer actually does

Each one depends on the one before it — matching without extraction is guesswork.

Extraction that keeps running

Scheduled, monitored pulls from each source system, including the old one with no API. A pipeline that fails silently is worse than no pipeline, because people keep trusting the numbers.

Matching and master data

Deciding when two records are the same customer, product or site, and which source wins when they disagree. The rules are a business decision, and they need an owner who can make it.

History that is kept

Source systems overwrite. A warehouse keeps what a record looked like last quarter, which is the only way to answer why a number has moved.

One place to read from

Reporting, forecasting and AI all read the same assembled copy, so they cannot quietly disagree — and none of them are querying a production database at month end.

Quality checks with teeth

Rules that flag the broken rows on arrival and tell someone, rather than letting them surface in a board pack. Most data quality failures are old, small and repeated.

The honest constraint

This layer cannot fix data the source systems keep breaking. Where a field is entered inconsistently at the point of capture, the durable fix is upstream — the warehouse only makes the cost of it visible.

A designer reviewing a website layout in Figma on a large monitor

Do you actually need a warehouse?

Often not yet, and we will say so. Two systems with decent APIs and modest volumes are well served by direct connections and a small reporting database.

A warehouse earns its place when several systems must be reconciled, history gets overwritten, reporting slows your systems, or the connections no longer fit on a whiteboard.

The same customer exists in five systems, with five spellings and no shared identifier.

What does data integration cost?

Cost follows how much your records disagree, not data volume.

Start with a $9,000 workshop

We look at your systems, IDs and questions, then tell you whether you need a set of connections or a real warehouse.

Disagreement drives cost

Storage and compute are cheap. The effort is reconciling records that describe the same thing differently.

You own the matching rules

Someone on your side must decide which records are the same customer. Without a named owner, projects stall here.

Amplify, then Apex

A first data layer is a fixed-price Amplify project. As new sources and rules keep arriving, it runs as an Apex partnership.

How a data project runs

Find out how bad it is (2–3 weeks)

We profile the actual records rather than the schema: how many duplicate customers, how many blank required fields, how far the systems disagree. The answer is worse than expected roughly every time, and it is better known now.

Two CodeFish team members in discussion across a meeting room table, one taking notes

One source, end to end (4–6 weeks)

A single system extracted, matched and landed with monitoring and quality checks in place. Doing one properly sets the pattern the rest follow, and it exposes the identifier problems early.

A designer sketching interface wireframes on paper beside a laptop

Add sources, then build on it

The remaining systems join one at a time, and the reporting, forecasting and automation work starts reading from the shared layer instead of from copies.

Four CodeFish team members working through a problem around a table of laptops, CodeFish Studio sign on the wall

Data questions we get asked

Not until several systems have to be reconciled rather than just connected, history matters because your systems overwrite it, reporting load starts affecting the systems it reads from, or the number of direct connections between systems has grown past what anyone can keep track of. Below that, point-to-point integration and a small reporting database are cheaper and do the same job.

With matching rules and a decision about which source wins for each field. Some of it is automatic — exact matches on ABN, email or account code. The rest is a judgement call someone in your business has to make and stand behind. Expect to name an owner for this; it is the part that stalls projects when nobody has.

Usually. Direct database reads, scheduled file exports and screen-level extraction all work, in that order of preference. The constraint is rarely getting the data out — it is that older systems often lack the identifiers needed to match their records to anything else, which is found in the profiling stage.

Related, and not the same. Integration keeps two systems in step operationally — an order placed here appears there. The data layer assembles a consistent copy for reporting, analysis and AI. The overlap is the matching: both need to know which records correspond, and doing that work once serves both.

For anything that reasons over your own records, yes. A model given inconsistent data produces confident, inconsistent answers, and the failure is harder to spot than a broken report. The good news is that the data work is not AI-specific — it is the same foundation reporting and integration need, so it is not a cost that belongs to the AI project alone.

In your own cloud account, in the Australian region unless you tell us otherwise, with access controlled per role and logged. You own the infrastructure and the pipelines the same way you own the code, so nothing here creates a dependency on us to keep operating.

Ready to build something great?

Local partners, global standards – full service design and development.