Zuul and Spring Cloud Gateway – Comparison, Benchmarks, LoadTesting

Spring Cloud Gateway and Zuul are different projects from the Spring community aimed to provide a developer-friendly way of writing Gateway services. While a many of the Spring Cloud users aware of the Zuul project, S-C Gateway is relatively framework which Spring Web Flux (Project Reactor) and the new
SpringBoot2. You can refer the question which I asked some time ago in StackOverflow for differences.
I have been using Netflix’s Zuul for over two years now and I am so far happy with its performance. I am eagerly waiting to see the much-purported Zuul2 which we can expect anytime. But S-C Gateway intrigued me for two reasons. One, it is coming from the
spring community using the latest spring 5, its support for non-blocking APIs, WebSockets, SSEs, etc.

Author of S-C Gateway Spencer Gibb has provided a benchmark app if you would like to take a look.

Note: The Spring Cloud Gateway used for this test is a pre-release version and the post will be updated after the GA. So take results of SC Gateway as a pinch of salt

But I wanted to test (stress) the service to its maximum capacity using different embedded web servers and conditions. Even though our gateway services will
be set up in HA environment in production applications, we can never be 100% sure about the incoming traffic. It is always better to test them from time to time by changing the parameters and fine tune to meet our expectations.

Gatling is used as the load testing tool and our candidates Zuul and S-C Gateway are tested with different embedded web servers

The aim of this blog post is not to find the winner or to create a debate on which embedded web server is best. In realtime the there are multiple parameters which we need to take into consideration before choosing a one. Based on the business problems and system ecosystems each application is unique and it is impossible to find a silver bullet to solve all the technical problems. So based on your need choose the libraries and tools wisely which helps you.
However, I will try to give you a perspective about the two awesome libraries which can be used a Gateway/Edge Service along with some of the excellent web servers. Below is my Zuul configuration to route the requests to httpbin.org

zuul:
prefix: /api
ignoredServices: "*"
routes:
myservice:
path: /myservice/**
serviceId: myservice
url: https://httpbin.org/

And an equivalent of the same in S-C Gateway can be written like below using the Fluent API. Note: You can also do the same
in the configuration file, but I felt writing as a code is more developer friendly. It’s more of the personal choice.

@Bean
public RouteLocator customRouteLocator(RouteLocatorBuilder builder) {
  return builder.routes()
    .route(r -> r.path("/getip")
    .uri("http://httpbin.org:80/ip"))
  .build();
}

Our Gatling Test file looked like the below where I changed the parameters to run the tests

class GatlingTest extends Simulation {
  val httpConf = http
    .baseURL("http://localhost:8080")
  val scn = scenario("Sample")
    .exec(http("GetIP")
      .get("/getip"))
    .pause(100 milliseconds)
  setUp(scn.inject(rampUsers(50000) over(60 seconds)) .protocols(httpConf))
}

Below is the summary of the results and the extensive report can be found at the below link.

+================+===========+======+===========+==========+============+======================+=============+
| Type:Webserver | Requests  |  OK  | KO(Error) | Mean RPS | t < 800 ms | 800 ms < t < 1200 ms | t > 1200 ms |
+================+===========+======+===========+==========+============+======================+=============+
| Zuul:Undertow  | 10000/30s | 4211 |      5789 |  142.857 |          2 |                   28 |        4181 |
+----------------+-----------+------+-----------+----------+------------+----------------------+-------------+
| Zuul:Tomcat    | 10000/30s | 9612 |       388 |  169.492 |       2918 |                  568 |        6126 |
+----------------+-----------+------+-----------+----------+------------+----------------------+-------------+
| Zuul:Jetty     | 10000/30s | 9776 |       224 |  227.273 |       2559 |                  537 |        6680 |
+----------------+-----------+------+-----------+----------+------------+----------------------+-------------+
| SCG:Netty      | 10000/30s | 8164 |      1836 |  185.185 |          0 |                    1 |        8163 |
+----------------+-----------+------+-----------+----------+------------+----------------------+-------------+
| Zuul:Undertow  | 1000/60s  | 1000 |         0 |   16.393 |        927 |                   13 |          60 |
+----------------+-----------+------+-----------+----------+------------+----------------------+-------------+
| Zuul:Tomcat    | 1000/60s  | 1000 |         0 |   16.393 |        990 |                    9 |           1 |
+----------------+-----------+------+-----------+----------+------------+----------------------+-------------+
| Zuul:Jetty     | 1000/60s  | 1000 |         0 |   16.393 |        942 |                   13 |          45 |
+----------------+-----------+------+-----------+----------+------------+----------------------+-------------+
| SCG:Netty      | 1000/60s  |  881 |       119 |   16.393 |        877 |                    2 |           2 |
+----------------+-----------+------+-----------+----------+------------+----------------------+-------------+

Conclusion

Even though, the tests indicate x is better than y. It can be easily deceiving. Results are highly dependent on the environment, downstream services, etc. So before making decision
run the tests in your environment with your systems which will help you greatly. The code base used for this post can be found on Github. Detailed benchmarks results can be found in the benchmarks folder inside the repository.

Update 1st August 2018

With the release of Spring Boot 2 and Spring Cloud 2 (Finchley Release), a lot of things have changed. Particularly in terms of performance. I re-run the tests again under the same environment and it was clear and plain to see that Spring Cloud Gateway has a superior performance compared to that of Zuul. Thanks to the Netty’s eventloop which powers the S-C Gateway and default web server for the springboot2. The updated code is available in the previous link.

+----------------+---------------+-------+-----------+----------+------------+----------------------+-------------+--------------+-------------+-------------+
| Type:Webserver | Ramp Up Users |  OK   | KO(Error) | Mean RPS | t < 800 ms | 800 ms < t < 1200 ms | t > 1200 ms | Mean T in ms | Max T in ms | Min T in ms |
+----------------+---------------+-------+-----------+----------+------------+----------------------+-------------+--------------+-------------+-------------+
| Zuul:Tomcat    | 10000/30s     |  4454 |      5546 |  142.857 |         13 |                   37 |        4404 |        10949 |       58921 |           0 |
| SCG:Netty      | 10000/30s     | 10000 |         0 |  277.778 |       8106 |                  692 |        1202 |          731 |        4966 |         270 |
| Zuul:Tomcat    | 5000/60s      |  4999 |         1 |   52.083 |         37 |                   68 |        4894 |        20308 |       54634 |           0 |
| SCG:Netty      | 5000/60s      |  5000 |         0 |   78.125 |       4426 |                  139 |         435 |          476 |        4956 |         259 |
| Zuul:Tomcat    | 1000/60s      |  1000 |         0 |   16.393 |        993 |                    6 |           1 |          346 |        1833 |         269 |
| SCG:Netty      | 1000/60s      |  1000 |         0 |   16.393 |        985 |                   12 |           3 |          346 |        1517 |         265 |
| SCG:Netty      | 30000/60s     | 30000 |         0 |  422.535 |      20252 |                 4048 |        5700 |          921 |        5802 |         269 |
| SCG:Netty      | 50000/60s     | 50000 |         0 |  381.679 |      34709 |                 6462 |        8829 |          905 |       17833 |         269 |
+----------------+---------------+-------+-----------+----------+------------+----------------------+-------------+--------------+-------------+-------------+

Leave a Reply

Your email address will not be published. Required fields are marked *