Cannot run agoo benchmark

[ioquatix]

koyoko% ruby ./agoo_bench.rb
Traceback (most recent call last):
./agoo_bench.rb:6:in `<main>': undefined method `new' for Agoo::Server:Module (NoMethodError)

[ioquatix]

Maybe it's related to ohler55/agoo#20

[ohler55]

Agoo changes as of version 2.0.0 so that the server is the Agoo::Server module. I have some examples on how to start I with and without rackup. For basic Rack with no static asset serving it is just

#config.ru
class FlyHandler
  def call(req)
    [ 200, { }, [ "flying fish" ] ]
  end
end
Rack::Handler::Agoo.run(FlyHandler.new)

Start with

ruby -r agoo config.ru

Of course there are additional options to run but that is the basic.

[ioquatix]

• So it's not possible to have multiple servers?

• Can you make something similar to falcon serve which loads the config.ru file? The config.ru you give above won't work with other servers.

[ioquatix]

Oh, I guess I can use rackup --server agoo?

[ohler55]

yes, you can. I'll post a revised benchmark file here in a moment.

[ioquatix]

I thought I'd give you some additional feedback.

Running the benchmark middleware (hello.rb)

• Agoo with something similar to agoo_bench.rb had amazing performance with something like 130,000 requests/s. It's not the best I've seen, considering native implementations of web servers, but it's certainly very good.

• Falcon had about 60,000 requests/second running the config.ru file. Using htop shows it using all 8 cores. Falcon is pure ruby by the way, so this is no small achievement.

• Agoo (using rackup) only had 9000 requests/second using the config.ru file. Using htop makes it look like only using one core. This is a similar limitation to falcon when using rackup so I'm sure we can improve this.

[ioquatix]

I'd suggest that we try to test all servers on a level playing field, using rackup. We should be able to provide tuning options through rackup. It should have good defaults.

[ohler55]

Agoo does not yet have clustering. That is in the works but not there yet. I know that is needed to scale the Ruby calls. Agoo has raw speed a single core down fairly well but it lacks the clustering option of Falcon. You must have a nice machine to have 8 cores 😄

I am fine using rackup but I'd also like to test the static file options for those servers that support it. Agoo now has an option to specify assets directories so even with rack it can serve static assets. For benchmarking it would be nice to show.

• straight rackup with clustering
• rackup with out clustering
• static file serving if the server supports that

I think arguments could be made for each of those modes. What do you think?

[ioquatix]

Only you can make a decision what works best in terms of options.

The defacto standard for static assets is to use public/. It's the default for passenger, puma, nginx+passenger, etc.

There is very little reason to change it IMHO, but having it as an option wouldn't hurt.

rackup is kind of shitty.

It works, and you should support it, but I don't know if you can expect great performance from it.

rackup inserts the lint middleware which makes it slow by default.

I don't think it's stupid that agoo executed by itself defaults to serving the application in config.ru in the current directory. I should probably make that the default in falcon too.

[ohler55]

I'm a bit surprised you are only getting 9K req/sec though. Are you ubuntu or OS X? OS X is pretty poor for IO. I'll have to check and see what is going on. I was getting 100K with Ubuntu earlier.

[ioquatix]

I agree that macOS has a shitty IO layer.

I was running on the same linux box with (4 core / 8 hyper threads). The upper limit with a simple TCP epoll server is around 400,000 connections per second. epoll doesn't scale linearly unfortunately. Single core performance can be more like 150,000 connections per second.

[ioquatix]

I made a config.ru file similar to the puma config, and then used rackup to run it with agoo, because I think this better reflects real world baseline performance. Obviously in this case there are some performance issues since involving agoo directly is much better performance.

[ohler55]

Is the default /public or ./public? I could easily have Agoo default to that for rack. Rails uses something different and Agoo detects and provides those.

Maybe it is rackup that is slowing things down. I'd prefer not to have that be the bottleneck when running benchmarks. How about we keep the benchmarks focussed on the servers and leave rackup out of the equation.

[ohler55]

Maybe some explanation or comparison between rackup and without?

[ioquatix]

The default is File.expand_path('public', __dir__).

I'm pretty sure that is also the default for rails - if you make your own rails website it will make a public/ directory.

[ohler55]

Are you using perfer or wrk? At those rates either is fine. Just curios what the setting were for threads, connections, and keep-alive are.

I believe Rails serves assets from the directories listed in Rails.configuration.assets.paths doesn't it?

[ioquatix]

Rails by default serves public assets from public/.

[ohler55]

Wow, your are so right about the rackup overhead. Without rackup I get 100K on a 4 core laptop. 10K with rackup.

[ohler55]

Benchmarks show the performance of the slowest link in the chain. Using rackup all we are doing is testing middleware times the number of nodes in the cluster. While it is important to point out which servers support clustering it doesn't give the total picture. I think hitting benchmarks for the three conditions I mentioned and then a nice explanation of how it all fits together would give a pretty clear picture of how all the servers stack up under various conditions.

[ioquatix]

There are some frustrating road blocks in your path w.r.t. real world performance and agoo. I like what you've done. So, I'm not criticising that at all.

Real Ruby applications don't scale up that well. The main issue for the server implementation is the GVL, but another part of that problem is blocking IO and the typical thread-per-request model.

I've started trying to address some of these issues here:

• Making fiber context switch ~2.2x faster: https://bugs.ruby-lang.org/issues/14739

• Transparent non-blocking IO: https://bugs.ruby-lang.org/issues/14736

These changes indirectly benefit you because then you can use your reactor/selector design with existing Ruby code with no changes required to that code.

I'd appreciate if you can comment on the 2nd issue above, mention how agoo implements it's own selector, and how such a design would benefit agoo. Imagine if you could transparently lower blocking operations into your own selector/reactor. It would be good, no?

[ohler55]

I'll have to check public with rails. I've just been looking at the variable up till now an didn't check to see where all the directories actually were.

[ioquatix]

Benchmarks show the performance of the slowest link in the chain. Using rackup all we are doing is testing middleware times the number of nodes in the cluster.

Don't try benchmarking a real rails application then - you are going to find out some painful limits w.r.t. the slowest link in the chain :p

[ohler55]

I did do some benchmarks with Rails. Agoo was faster serving generated content than the default but only by 50% or so. It was 2000 times faster serving static assets though.

Oh, second point. That would be the config.ru file. Yeah, it should work with just run as long as you have r agoo -s agoo on the command line.

[ioquatix]

I did do some benchmarks with Rails. Agoo was faster serving generated content than the default but only by 50% or so

I'd be surprised by this result, if you only use one CPU core. The majority of the time should be in Rails code, not agoo.

I think you'd find that any web server that can run multiple processes, with decent single-core performance would be better than a single core solution.

[ohler55]

Agoo does use multiple threads but only with the GVL any more than one Ruby eval thread doesn't help that much. There is that option in Agoo but Agoo does not yet fork to run multiple separate Ruby interpreters. I assume thats what you are referring to when you say core, right. I missed that when you first mentioned cores. I was thinking CPU cores, not Ruby cores.

[ioquatix]

Sorry if my terminology was confusing.

I guess by n-cores I mean n processes which can run in parallel. You obviously need n CPU cores in order for this to occur. You can still run 16 processes on 8 cores, but you'll end up with pre-emptive multi-tasking.

For Ruby, running n-cores is preferable to n-threads because Ruby threads are concurrently scheduled when running Ruby code which makes them bad w.r.t. physical CPU utilisation.

[ohler55]

That has been my experience with Ruby threads as well. The only downside to multiple cores is the app has to be stateless but I suppose Rails and Rack users have accepted that by now and rely on a database for all state.