Many people believe that a successful Big Data & AI platform requires an army of developers, DevOps engineers, system administrators, database administrators, many specialist for real-time integration with other systems, data analysts, data scientists, and many others, as well as a large team to maintain and support such systems — which includes monitoring, patching, upgrades, and so on.
Similarly, there is a common belief that Big Data and AI architecture must have an extremely complex SW architecture, based on Kubernetes and the Cloud, in order to scale during peak load.
None of that is true.
Team of two people, my colleague Snježana Ovničević who was responsible for mapping functional requirements, and me who made technical part (software architecture, real-time integration, installation and administration of Big Data and AI components, as well as system development and optimization), was enough despite the huge amount of data we are face with.
We got the chance to build a new Big Data platform quite unexpectedly, with a very small budget that we spent on internal resources (memory, storage, and CPU) – there was no spending on commercial software or subscriptions to Cloud services.
Along the way, we also solved many other goals such as:
- Easy to scale up & out
- Flexibility – we can run Big Data & AI on-premises, on Kubernetes or in the Cloud
- Simplicity – Simplicity in IT means very low maintenance costs, highest possible speed, low number of incidents, fast and easy troubleshooting, low resource consumption and price efficiency
- Speed – we are not only ensuring real-time integration capabilities, but also real-time analytics, and real-time AI with low latency
- No external vendor dependency/costs – we do net rely on external vendors or Cloud vendors. We control the whole process which is fully developed in-house
- No commercial software components – – The whole architecture is based on FOSS (free and open source) components. There are no commercial software components we depend on.
- Resource efficiency
- Price efficiency/cost effective – It’s very simple to calculate 5-year TCO (total cost of ownership) and to do ROI (return of investment). For large enterprises, predictable costs are paramount.
- Data Sovereignty – For this BigData & AI platform,we have a full control of highly sensitive GDPR related data, and all data remains within HT Data centers within Croatia borders.
- Security
- Self-contained / No vendor lock–in – new architecture does not depend on any commercial software or Cloud provider, whose goal might be to lock our own data, once we migrate our data to commercial software or Cloud vendor.We are free to do whatever we want with no additional costs.
- Real-time integration
LinkedIn link:
https://www.linkedin.com/pulse/celebrating-first-year-new-big-data-ai-platform-josip-pojatina-fhlof
Comments