- the collector of events

Overview

This collector meant to be a highly available RESTful web service that receives events from client devices and secures them agnostic of cloud targets.

The collector is implemented in Go, runs on Ubuntu and OSX.

Events are sent in Protobuf messages.

Currently supported cloud targets are (tested throughput on a c3.xlarge computer with 4vCPU in AWS):

• Amazon Web Services Simple Notification Service: 59k events/minute, 70 multi payload requests/s
• Amazon Web Services Simple Storage Service: 6.2M events/minute, 8k payload requests/s
• Microsoft Azure Blob Storage: 6.05M events/minute, 7.8k multi payload requests/s
• Microsoft Azure Queue Storage: 5k events/minute, 5 multi payload requests/s

6Wunderkinder used a similar node.js based service that secured messages in AWS SNS. Based on experiences we've rewritten the app in Go that can handle 100x more requests on equal hardware resources.

Inspired by UNIX philosophy (do one thing and do it well) and Marcio Castilho's approach.

Clients

No official client is available at the moment. If you want to write your own please check out our pseudo client specification.

Installation

Please install Go 1.5+ and Python 2.7 or 3.3+.

$ sudo make install/go && source ~/.profile # you can install golang with this on OSX/Ubuntu if you need it
$ sudo make install/protobuf # initialize communication format
$ make install/pkg # golang dependencies
$ sudo make install/wrk # install http benchmarking tool
$ make install/utils # utils for development

After the package installation, please create your configuration file based on the sample configuration.

Set up your environment variables.

export HAMUSTRO_CONFIG="config/yourconfig.json"
export HAMUSTRO_HOST="localhost"
export HAMUSTRO_PORT="8080"

Start collector

You can start the server for development with the following command:

$ make dev

In the development mode it provides useful messages to track what's happening within the collector. Furthermore it notifies the clients with JSON responses on error.

To turn off the notifications and run the collector for production, please use the following command:

$ make server

Tests

You can run the tests with

$ make tests/run

If you want to start a stress test, please use

$ make tests/stress/n # 1-25 payloads/request
$ make tests/stress/1 # 1 payload/request

During the stress test, you can profile the heap/cpu/goroutine usage easily in development mode, just type

$ make profile/heap
$ make profile/cpu
$ make profile/goroutine

License

Copyright © 2016, Bence Faludi.

Distributed under the MIT License.