{"id":932,"date":"2020-04-08T18:49:36","date_gmt":"2020-04-08T10:49:36","guid":{"rendered":"http:\/\/cleardatascience.com\/?p=932"},"modified":"2020-04-09T10:13:41","modified_gmt":"2020-04-09T02:13:41","slug":"build-your-own-best-fit-data-repository","status":"publish","type":"post","link":"https:\/\/cleardatascience.com\/en\/build-your-own-best-fit-data-repository\/","title":{"rendered":"Build Your Own \u201cBest-fit\u201d Data Repository"},"content":{"rendered":"

CDS technical team is always using a mixture of commercial licensed and open source software together for our solutions and professional services to clients.<\/p>\n

With the rapid development of open source technology, there are more choices available to build data warehouse and \/ or Big data platform.\u00a0 \u00a0There are several different real application examples available with our technical team members:<\/p>\n

    \n
  1. Modern Data Warehouse<\/li>\n
  2. NoSQL Database for IoT<\/li>\n
  3. Big Data Repository<\/li>\n
  4. Data Lake<\/li>\n<\/ol>\n

    Modern Data Warehouse<\/strong><\/p>\n

    In this year, we have used PostgreSQL database to build a modern data warehouse for an asset management company.\u00a0\u00a0 The PostgreSQL database is running on a cluster with SSD storage for the speed to store the commodities\u2019 prices and transaction history. \u00a0\u00a0For the modern data warehouse, it is better to have the possibility to integrate with Apache Cassandra \/ Hadoop for semi-structured or even unstructured data as a future proven solution and details available in the later part of this article.\u00a0 PostgreSQL is a very good solution and also the MariaDB is another possible candidate with Enterprise class technical support subscription services available.\u00a0 However, the latest version MariaDB is no longer official supported Apache Cassandra NoSQL direct connection.<\/p>\n

    \"\"<\/p>\n

    A Typical Architecture for a Data Warehouse<\/p>\n

    NoSQL Database for IoT<\/strong><\/p>\n

    We are helping a courier corporation to track their tracks on the road with a GPS sensor module for every truck.\u00a0 Then, the route for each truck is being recorded and upload to an Apache Cassandra NoSQL database for further analysis for the performance of each individual driver and the traffic conditions.\u00a0 For NoSQL database, we are also using Redis for storing temperature and humidity information for different data center or computer lab owners for the air-conditioning monitoring and auto-adjustment by system integration with their air-conditioning system.<\/p>\n

    \"\"<\/p>\n

    GPS Sensor<\/p>\n

    Big Data Repository<\/strong><\/p>\n

    In order to handle large amount of unstructured data, one of the most efficient way is to store it into the Apache Hadoop.\u00a0 We are helping different clients mostly retailers to store multiple social media logs to apply sentiment analysis for better customer services.\u00a0 They are feeding the unstructured data from different social media to the HDFS.\u00a0 Then, they are streaming the data with Spark to do near real-time analytics.\u00a0 This is important to have prompt responses to the customers both online & offline, then leading to better answers in logistics and inventory management.<\/p>\n

    \"\"<\/p>\n

    Big Data Repository Design – available for near real-time analytics<\/p>\n

    Data Lake<\/strong><\/p>\n

    For a large scale corporation, it is possible for them to have hundreds of systems and data silos with different owners.\u00a0 It is difficult to analyze their data as a whole.\u00a0 There are several options to build a data lake:<\/p>\n