Normally, architecturally, we do not see combining analytics and transactional data as a pattern to repeat. But frankly speaking, for some systems it is possible to start with something lightweight, easy to maintain and easy to deploy the first time, because it minimises time to money and time to production.
A fair start that becomes debt
It's okay to start with an easy implementation, keeping in mind that it should be reorganised architecturally when the time comes and the traffic gets bigger. The problem is that in this case it becomes technical debt, and technical debt, as usual, has lower priority than high-priority business tasks. It can be postponed for a long time.
Still, from my point of view, it's okay to start with something easy: first, to deliver the product as fast as we can, and second, to understand what the requirements for analytics will be. When you start an MVP, a first version that will change dramatically after testing or after the first users, it's worth thinking about combining everything in one good architecture, but still a simple solution.
Some runtime or daily analytics may also stay in the same system, because you need them at the moment the issue occurs. A small amount of analytics for daily usage and daily decisions is okay in the main system, from my point of view. But nowadays there are a lot of systems and packages that let you replicate the data you need for analytics very fast, even almost in real time, and analyse it in real time. So the approach of keeping analytics in the main system is becoming more and more legacy.
Your team sees the moment first
The decision to split usually relies on a forecast of the customers' transactional activity, the length of the processes and the workload the reporting process generates. Usually the team and DevOps see this moment very clearly. The most important thing is to prioritise it: to give them time for the technical debt and the reorganising work when the time comes.
From outside, we usually see signs like a very long generation of reports, or performance degradation during the reporting period. But usually the team knows better, and sees the earlier signals that you have to change the strategy of implementation.
You cannot say it in numbers. It depends on the performance and the specifics of the processes implemented in the system, so there is no universal advice.
Split when your team says it is time
If reporting is starting to slow your transactional system, we can look at what to split out first.
Split, rather than rebuild
In one project, we ran the reconciliation of transactions and built the reports at night, so as not to load the main service. But often the reconciliation depended on how fast the external systems responded as well as on our system, and did not finish within the allotted window. The analysts also asked for access so that they could see aggregated metrics all the time. That meant three or four heavy processes competing for one system. On top of that, it brought the risk of several roles working on the production system, roles that should not affect how the core runs. The cure, of course, is to split them apart.
My approach here is to rely on the working system, because first of all you know its bottlenecks, and you take them to a new system or a new process. If the performance degradation is not fixed after this reorganisation, then for sure you have to think about refactoring the whole system, or creating a new one. But usually it's enough to split.
When you start everything from the inputs and from a system change, you will generate new issues and new problems that did not occur in the previous system. You will cure one thing and break another. It can also happen that you don't have the time or the resources for the task that is the priority right now, because you are busy with a system change that is not necessary. But it depends on the flexibility of the initial architecture of the current system.
Two kinds of history
As a rule, when a data warehouse is built today, we separate the incoming flow, real-time processing, dashboards, the analytical layer, staging and so on. We don't drag everything into one.
For quarterly reports there is a long history in staging, built with views for the slices you need. For those who work in real time, a mobile bank for example, there is a small slice and a short history: a lot is kept in memory and in an intermediate layer for the critical things, and synchronised with the core banking system where it is necessary.
Check the data before the analytics
The first question is usually about the quality of the data, and how current it is. We sometimes see that analytics requirements cannot be met properly, with high quality, if the initial data is rubbish. So we usually suggest assessing the quality of the data first. After that, we can suggest fixing it, improving it, or building on it, but with a continuous improvement process. And after that, think about analytics, because garbage in, garbage out.
Before you commission analytics
- Assess the quality of the data, and how current it is, before any analytics work.
- Ask your team and DevOps whether the moment to split is coming. They see the signals before reports get slow.
- When they say it is time, give the split its own priority. Left as technical debt, it waits behind business tasks.
- Split the analytics out first. Think about refactoring the whole system, or a new one, only if the performance degradation stays after the split.
Check the data first
Tell us what your analytics has to answer. We will begin with the quality of the data behind it.
Can analytics stay in the transactional system?
For some systems, yes, as a start. Starting with something lightweight minimises time to money and time to production, and helps you understand what the analytics requirements will be. A small amount of analytics for daily decisions can stay in the main system, although replication that works almost in real time is making this approach more and more legacy. Plan to reorganise it when the traffic grows: until then it is technical debt, and technical debt usually waits behind business tasks.
How do you know it is time to split analytics out?
There is no universal number: it depends on the performance and the processes in the system. The decision usually relies on a forecast of customer transaction activity, the length of the processes and the load that reporting generates. The team and DevOps usually see the moment first, and the most important thing is to give them time for the work. Seen from outside the team, the signs are reports that take very long to generate, or performance degradation during the reporting period.
Do you have to rebuild the system to get proper analytics?
Usually it is enough to split: rely on the working system, whose bottlenecks you already know. Rebuilding everything, from the inputs to the system itself, generates new problems that the old one did not have, and takes time from the task that is the priority now. If the performance degradation stays after the split, then it is time to think about refactoring the whole system or building a new one. Whether you can rely on the working system depends on how flexible its architecture is.
What should be checked before an analytics project?
The quality of the data and how current it is. If the initial data is rubbish, the analytics requirements cannot be met with high quality: garbage in, garbage out. Assess the data first, then fix it, improve it or build on it with a continuous improvement process, and after that think about analytics.

