Use of ByteBufferPool

[vgkholla]

Creating this issue to start an initial discussion on and list places where the ByteBufferPool can be used. Some of the tasks will require issues of their own.

P.1) Improve the current implementation of ByteBufferPool (SimpleByteBufferPool) or create a new implementation that actually pools buffers. This will require design documentation and discussions.
P.2) Use the ByteBufferPool (this is independent of P.1) in the NonBlockingRouter while filling chunks and also in BoundedByteBufferReceive. Using it in BoundedByteBufferReceive might require a bit of effort but it has to be done for pooling to be correct and effective since GETs are expected to be a large chunk of traffic in any use-case involving Ambry.

[haggag]

Consider using Netty's ByteBuf and its jemalloc implemention to support buffer pooling. https://netty.io/4.1/api/io/netty/buffer/ByteBuf.html