Photo by Chris Liverani on Unsplash
Introduction
Introduction
When developing modern applications, performance is a crucial consideration, particularly when interacting with databases. The choice of technology for managing database access can greatly impact the speed, scalability, and resource efficiency of your application.
This post compares four approaches for handling database operations: Hibernate, Hibernate Reactive, Hibernate with Virtual Threads (introduced in Java 21), and R2DBC (Reactive Relational Database Connectivity). The aim is to assess their performance in typical application scenarios, considering factors like throughput, latency, and resource utilization.
Now, let’s dive into the definitions and features of each of them to better understand how they work before examining their performance.
Overview of the Approaches
Overview of the Approaches
Hibernate
Hibernate
Hibernate is a long-standing ORM framework that simplifies database interactions by managing object-relational mapping. It uses synchronous, blocking I/O operations, making it reliable but potentially less scalable, especially when dealing with a large number of database operations.
Hibernate Reactive
Hibernate Reactive
Hibernate Reactive extends Hibernate by incorporating non-blocking, reactive programming principles. It allows applications to handle more concurrent requests without the limitations of thread blocking, which can improve scalability and responsiveness.
Virtual Threads
Virtual Threads
Virtual Threads, introduced in Java 21, offer scalability and concurrency. By improving thread management, they allow traditional blocking code to perform more efficiently, offering an alternative to reactive models without requiring a complete change in programming style.
R2DBC
R2DBC
R2DBC is designed for non-blocking, reactive access to relational databases. It works decently for applications requiring asynchronous data access, providing lower latencies and better handling of concurrent operations compared to traditional synchronous approaches.
Performance Testing Approach
Performance Testing Approach
For the performance testing, I used Locust, a load-testing tool, to simulate various levels of user activity and interactions with the database. The reason for using Locust was to observe the changes in requests per second (RPS) and response times under different load conditions, providing insights into how each approach scales and responds to increasing traffic.
To examine the performance of each approach under different types of database operations, I tested three key methods:
- Find all — retrieves all records from the database.
- Select by ID — fetches a single record based on a given ID.
- Create User — inserts a new user into the database.
Additionally, I applied pagination to the “find all” method to measure how each approach handles large datasets. One important detail is that repeatedly fetching data with the same ID might cause Hibernate’s cache to interfere with results, potentially making Hibernate seem faster than it is.
To introduce some delay in database operations, I also tested with pg_sleep(1), which adds a 1-second delay to simulate long-running queries or network latency. This helps in assessing how each approach behaves when facing slower database responses. Additionally, it was used to observe how the pg_sleep affects the RPS (request per second).
Lastly, I monitored CPU and memory usage during the tests to evaluate resource efficiency, ensuring a comprehensive performance comparison across all approaches.
Performance Test Results
Performance Test Results
All tests were conducted using Java 21, with PostgreSQL version 15.8 as the database. The environment for these tests was a 2020 M1 MacBook Air with 8GB of RAM.
Hibernate
Hibernate
Number of users(peak concurrency) : 8000
Ramp up(users started/second): 50
CPU and Memory Usage for 8000 users:
Number of users(peak concurrency) : 8000
Ramp up(users started/second): 50
With pagination applied to find all methods:
Number of users(peak concurrency) : 8000
Ramp up(users started/second): 50
With using pg_sleep(1):
Hibernate Reactive
Hibernate Reactive
Number of users(peak concurrency) : 8000
Ramp up(users started/second): 50
CPU and Memory Usage for 8000 users:
Number of users(peak concurrency) : 8000
Ramp up(users started/second): 50
With pagination applied to find all methods:
Number of users(peak concurrency) : 8000
Ramp up(users started/second): 50
With using pg_sleep(1):
Hibernate with Virtual Threads
Hibernate with Virtual Threads
Number of users(peak concurrency) : 8000
Ramp up(users started/second): 50
CPU and Memory Usage for 8000 users:
Number of users(peak concurrency) : 8000
Ramp up(users started/second): 50
With pagination applied to find all methods:
Number of users(peak concurrency) : 8000
Ramp up(users started/second): 50
With using pg_sleep(1):
R2DBC
R2DBC
Number of users(peak concurrency) : 8000
Ramp up(users started/second): 50
CPU and Memory Usage for 8000 users:
Number of users(peak concurrency) : 8000
Ramp up(users started/second): 50
With pagination applied to find all methods:
Number of users(peak concurrency) : 8000
Ramp up(users started/second): 50
With using pg_sleep(1):
Test Result Analysis
Test Result Analysis
The performance tests conducted using Locust revealed significant differences in response times and resource usage among the four approaches.
Response Times
Response Times
Hibernate with Virtual Threads consistently exhibited the lowest response times, showing their efficiency in handling I/O-bound operations. Their lightweight, scalable concurrency model enables them to manage numerous tasks simultaneously, resulting in faster response times even under heavy loads.
Following closely were Hibernate Reactive and R2DBC, both leveraging non-blocking I/O to enhance performance. While Hibernate Reactive performed admirably, it trailed slightly behind Hibernate with Virtual Threads in terms of response time. The non-blocking I/O architecture allows multiple operations to be processed concurrently, minimizing wait times for I/O operations like database queries. Consequently, Hibernate Reactive outperformed Hibernate, particularly in terms of response time.
In contrast, Hibernate relies on a blocking I/O model, where each thread must complete its database operation before proceeding to the next task. This sequential processing introduces delays and increases latency, especially in high-concurrency scenarios. The blocking nature of Hibernate explains its higher response times, making it less effective than the more scalable alternatives.
Impact of pg_sleep(1)When testing with pg_sleep(1), all four approaches experienced a significant drop in RPS as expected (compared to test results without pagination). This delay simulated slower database responses, and the results clearly showed that adding latency had a substantial impact across the board. Each approach, regardless of its architecture, saw a notable decrease in throughput when subjected to this artificial delay.
Effect of Pagination
Effect of Pagination
When pagination was applied to the “find all” method, Hibernate performed worse in terms of response time, as expected. Pagination increased the complexity of data retrieval for Hibernate, indicating its blocking I/O model’s limitations. This performance drop was anticipated, as pagination introduces additional processing overhead in blocking architectures. In contrast, the reactive approaches, especially Hibernate Reactive, handled pagination more gracefully, maintaining better performance due to their non-blocking nature.
CPU and Memory Usage
CPU and Memory Usage
There were also notable differences in CPU usage. R2DBC and Hibernate consumed significantly more CPU resources due to the overhead associated with managing blocking threads and context switching. Conversely, both Hibernate with Virtual Threads and Hibernate Reactive demonstrated far more efficient resource usage, resulting in lower CPU consumption. This efficiency not only contributes to their speed but also makes them more scalable and cost-effective for high-throughput applications.
In terms of memory consumption, Hibernate with Virtual Threads and Hibernate exhibited the highest memory usage, while Hibernate Reactive stood out as the most memory-efficient option.
Failure Rates Under Load
Failure Rates Under Load
Under high load conditions (specifically when simulating 8,000 users in Locust) both Hibernate and Hibernate Reactive experienced a significant increase in failure rates. In contrast, Hibernate with Virtual Threads maintained stable performance with no notable rise in failure rates, demonstrating their resilience under heavy load. This further underscores the effectiveness of Hibernate with Virtual Threads in high-concurrency environments.
As the number of users exceeded 6,000 and the requests per second rose above 150, a significant increase in failure rates was observed across all tested frameworks, leading to errors such as ConnectionResetError and ConnectTimeoutError. This emphasizes the need to implement robust error handling and connection management strategies on the client side to handle these scenarios better. On the server side, the primary approach involves leveraging autoscaling based on key metrics like error rates, response times, and resource utilization to maintain stability and performance as demand increases.
Key Performance Metrics at 4000 Users: Comparative Table of Results
Key Performance Metrics at 4000 Users: Comparative Table of Results
In addition to these results, I wanted to highlight a key milestone by showing the results in a table. I believe this will help everyone better understand the performance outcomes.
At 4000 users:
Based on these tables, Hibernate with Virtual Threads consistently delivers the best performance across all user loads, achieving the highest RPS. In contrast, Hibernate shows the poorest performance, with the lowest RPS. Both R2DBC and Hibernate Reactive perform better than Hibernate, though their performance still falls short of Hibernate with Virtual Threads. Regarding CPU consumption, Hibernate with Virtual Threads again demonstrates the best performance, while R2DBC shows the highest CPU usage. However, in terms of memory usage, Hibernate with Virtual Threads performs the worst, whereas Hibernate Reactive demonstrates the best memory efficiency. Therefore, it’s clear that each approach has its trade-offs, making it essential to choose the one that best aligns with your project’s specific requirements.
Conclusion
Conclusion
In this performance comparison, Hibernate with Virtual Threads, Hibernate Reactive, and R2DBC outperformed traditional Hibernate in terms of response times. Hibernate with Virtual Threads stood out as the top performer, offering lightweight, scalable concurrency. Hibernate Reactive also demonstrated strong performance, leveraging non-blocking I/O to handle concurrent operations efficiently. While R2DBC performed well, it was closely aligned with Hibernate Reactive in terms of speed.
Moreover, hibernate showed its limitations with blocking I/O, resulting in higher response times and resource consumption, especially under heavy load and when dealing with pagination. The introduction of pg_sleep(1) further highlighted the performance bottlenecks in all approaches, but especially in blocking models like Hibernate.
Ultimately, choosing between these approaches depends on your specific application requirements. If your system demands high concurrency, scalability, and low-latency operations, Hibernate with Virtual Threads or Hibernate Reactive are strong contenders. On the other hand, if your application doesn’t need high concurrency and prioritizes stability, traditional Hibernate may still be a viable option. Each approach offers distinct advantages, and understanding the trade-offs between blocking and non-blocking models can help developers make informed decisions to optimize performance.
