Reporting in Microservices: How To Optimize Performance
Microservice architecture is becoming increasingly popular among app developers, thanks to its flexibility and scalability that allows for updates and new features to be added without overhauling the whole system. However, the use of separate user interfaces, databases, and servers for each microservice makes app optimization a challenge, considering the diversity of programming languages and API security issues.
The Freshcode team faced the trouble of updating a legacy EdTech app running on MS SQL that used over 10,000 files created in ColdFusion and had dozens of microservices for disparate features. The app was intended for governments and education networks with hundreds of campuses and schools and had a full complement of features to appease both national universities and small schools. With conflicting microservice operations, reporting was a problem the FreshCode team had to tackle.
How To Improve Microservice Performance
Engaging different microservices caused a plethora of performance issues for the EdTech app, so we came up with an idea to develop a new reporting microservice. Instead of trying to reconcile conflicting technologies, the new microservice would get data directly from different databases, pass it through the messaging, and steam computation systems before storing it in a separate database. After that, the microservice could use the data from the database to create custom reports.
The task was further complicated by the need to develop two identical solutions (self-hosted and SaaS versions), so the team browsed several technology alternatives for every aspect of the reporting microservice to come up with the best stack for both deployment schemas.
Reporting Module Implementation in 6 Steps
The Freshcode team broke down the microservice development process into six stages. And as always, they started by analyzing requirements and selecting the best technologies and solutions.
1. Change Data Capturing
Oracle was the central reporting microservice database, so we went with StreamSets Data Collector as a customizable open-source solution with built-in Oracle CDC support. Matillion and Apache NiFi are equally user-friendly and beloved by developers, and they would have made the cut if they’d offered similar Oracle support.
2. Messaging System
The development team chose Apache Kafka as a durable, scalable, and fault-tolerant messaging system. Though it requires expert DevOps, the solution boasts native in-stream computations and batch mode. Besides, unlike AWS Kinesis, Apache Kafka can be used in-house.
3. Streaming Computation Systems
Despite its complicated architecture, Apache Flink was the best choice for the EdTech app thanks to its fault-tolerance, scalability, low latency, and exactly-once state consistency. AWS Kinesis Services were also among the top choices, but they couldn’t be used on-premise, so Freshcode had to forego the AWS solution.
4. Data Lake
AWS S3 was chosen to create a depository of historical and current data. It’s a durable and cost-effective open-source solution that’s easy to integrate with other AWS products. The Freshcode team considered Apache Hadoop as a worthy alternative, though it’s a more challenging option when it comes to deployment and management.
5. Report Databases
The reporting microservice had to work with other app modules, and the team suggested AWS Aurora as the best choice. It’s pretty fast for an SQL database while being highly scalable and durable. Though the minimally available instance is too big, it could be replaced with plain PostgreSQL. Other options the team considered included AWS Redshirt, Apache Druid, and Kinetica.
6. Report Microservice
The core features of the microservice included storing data objects and their relations, as well as generating custom reports based on the selected data objects.
Technological Stacks for SAAS and Self-Hosted Solutions
Freshcode developers settled on NodeJS for creating a microservice for the SaaS EdTech solution. It relied mostly on AWS products, like AWS S3, Aurora, and ElasticCache, complimented by StreamSets and Apache Kafka. This technology stack was ideal, as it could be easily replaced by self-hosted alternatives without disrupting the code and logic of the microservice for another deployment schema.
The in-house reporting microservice used Minio, PostgreSQL, and Redis instead as their APIs are fully compatible and provide seamless operation.
The Bottom Line: Custom Reporting in Microservices
Successful implementation of the reporting microservice, along with other improvements achieved by Freshcode, enabled the EdTech company to update the design and architecture of the app and make it better through the addition of new and enhanced features. Besides, by removing reporting performance issues, the app became more flexible and scalable for in-house and SaaS deployment.
If you want to learn more about the specifics of this project and the details of reporting microservice development with timeframe and budget estimates, check out the full EdTech case study.
