ZakariaLutfi (ZakariaLutfi)

2020: Near Realtime Load Data to MySQL Database

As a data analyst I have a need for the availability of large datasets. This data comes from many things such as HTTP Request, Database, and activity log and then I need to save it in an RDBMS. But I face a problem to do the process of recording data into a MySQL database. The data sent amounts to millions. I feel that this cannot be done simultaneously using standard SQL commands.
We decided to use Kafka as a tool to overcome the process queue. A lot of data is produced into Kafka topic. Then I created a worker process using python and SQLAlchemy which consumes messages from Kafka to be translated and inserted into the database. The reason for using python instead of using other tools that are more complex is that we don't want to deal with complicated things like installing tools and maintaining processes. So python is the best option we have.