PaaS benchmark: Clever Cloud vs Scalingo vs Hidora

This series of posts shares hands-on feedback on French PaaS providers, with one problem in mind: hosting a fleet of 120+ WordPress sites.

In this series:

• Part 1. Background to the French PaaS comparison
• Part 2. PaaS comparison: OVH falls overboard
• Part 3. PaaS comparison: Clever Cloud, the boat is taking on water the captain rights the ship
• Part 4. PaaS comparison: Scalingo stays afloat
• Part 5. PaaS comparison: Platform.sh lies at anchor
• Part 6. PaaS comparison: Hidora, the Swiss cloud
• Part 7. PaaS comparison: performance, Clever Cloud VS Scalingo VS Hidora

---

EDIT 01/21 The performance tests have been rerun. The first batch, from 2019, is available here: all ~80 tests from the 2019 PaaS performance comparison between Scalingo and Clever Cloud; it contained 80 tests. In 2021, I kept the tests much more concise; they’re available in this spreadsheet.

Measured performance

Included in this benchmark:

• ✅ Clever Cloud
• ✅ Scalingo
• ✅ Hidora
• ⛔️ OVH Cloud Web (couldn’t get the project running, see Part II)
• ⛔️ Platform.sh (too expensive for our WordPress use case)

The tests were run under real-world conditions. I took a “representative” WordPress site that’s already in production with us. These aren’t tests on a plugin-free WordPress with a single blank page.

I ran the tests with a load testing tool (k6.io) from a server in Paris, FR: 5 unique visitors per second for 2 minutes.

The tests were run under the same conditions on every PaaS: same versions of WordPress and the plugins, same page tested. No optimizations, except for tuning the pm.max_children variable to handle request concurrency in php-fpm.

Before recording the results, I spent some time finding the optimal value for the pm.max_children setting, which tells each scaler how many concurrent PHP requests it can handle.

Set it too high, and performance tanks because there isn’t enough RAM (it swaps). Set it too low, and the scaler is throttled, and so is your performance.

⚠️ Note that the number of unique visitors varies, even when the test is set to 5 unique visitors per second for 2 minutes. I never got 5x60x2 = 600 requests. So remember to look at the number of requests the server actually had to handle. ⚠️

Test results (excerpts)

5 clients per second over 2 minutes

Test setup

• The Bedrock edition of WordPress, which makes WP compatible with PaaS hosting, on top of providing modern tooling (configuration through environment variables, WordPress plugins managed with composer…)
• PHP 7.4.x
• Site 1
  • WordPress 5.6.x
  • List of dependencies and plugins: composer.json
• pm.max_children is set to 10