- The first step in laying the foundation of the big-data platform was to install and configure the infrastructure (servers, virtual machines). We opted for a scalable cloud-ready infrastructure, using the Hadoop platform.
- For the data collection/transfer/ingestion layer we chose Apache NiFi, an extremely flexible, efficient, and complete solution. We used Apache Hadoop (HDFS), on multiple nodes, for distributed data storage.
- Distributed data processing is made with Apache Spark and real-time events are processed with Apache Kafka (Kafka streams).
- Storage of data processed according to business needs is made using Apache Cassandra (high performance for scalability, throughput, and availability). For other usage scenarios, PostgreSQL, Redis, and Apache Druid are also used.
- Data is offered for consumption (ingestion) in different formats, via secure APIs.
- From the security point of view, differentiated access was configured between different data areas, alerts were configured, as well as data recovery procedures in case of emergency.
- The platform includes a monitoring solution offering information and alerts not only concerning the infrastructure and technical components, but also for the business processes being executed. For logging and monitoring, we used ELK Stack (Elasticsearch, Logstash, Kibana).
- The solution has been built so that any component can scale horizontally for future business needs or technological developments, offering, at the same time, high availability and fast access to data.
The big-data platform implemented for Carrefour gathers data from across 400 stores concerning 500.000 unique products, being updated and served to other applications in real-time. There are aggregated, among others, 35 million records that show the image of each product (stock, price, attributes, promotions, etc.) in all stores. Data flows processed in one day exceed 10 billion records. The platform helps streamline and simplify internal processes, ensuring real-time access to data for the entire Carrefour application ecosystem, thus becoming a unique operational data source. For Carrefour customers, the entire process translates into improving the shopping experience through the eCommerce platform and product delivery services (Bringo).
The Team
The first phase of the project started at the end of 2017. Subsequently, new flows and features have been constantly developed, the project still being in progress. The project team is a complex one, including Big Data Solution Architects, Big Data Developers, Big Data Engineers, SysOps, Project Manager.
Technologies
NiFi (for Data Ingest)
HDFS (for Data Lake)
Kafka Streams (for Event Sourcing)
Spark (for Batch Processing)
PostgreSQL, Redis, Cassandra Druid (for Data Storage)
RO
EN
