University of Twente Student Theses
Bringing scalability/failover to a complex producer/consumer implementation
Houtman, J. (2009) Bringing scalability/failover to a complex producer/consumer implementation.
![[img]](http://essay.utwente.nl/style/images/fileicons/application_pdf.png) |
PDF
10MB |
| Abstract: | This research centers around two subjects. First queues used in a producer/consumer paradigm and implemented using a database must be made selfscalable. Secondly a self-scalable system for the queue consumers and h-workers (programs that perform tasks like data preparation) must be developed. Two solutions were suggested. The rst solution creates a container from a xed set of processors or h-workers, when required a queue is also added. A container is then run on a node, eectively creating a consumer side queue. Containers only hold h-workers or processors that perform the same task, to scale capacity for a task more containers are created. The de-centralized nature of the solution only allows for gradual scaling. This is a very ill t for the h-workers, because they run in intervals and have a very abrupt need for resources. The second solution separates the queues from the consumers on a conceptual level. This allows dierent systems for the queues and the consumers/hworkers. Each queue is segmented into a minimum of two segments and divided over a set of nodes. A weight associated with each queue segment determines how the incoming events are distributed over the available segments. Queues are typed and dierent types can coexist on the same node. Incoming events are routed to appropriately typed queue-segments. Scaling takes place by changing the routing of events to be better spread over the available nodes. Consumers and h-workers are implemented in a master/worker model (further called manager/worker) which runs on top of a batch system, thereby allowing the manager to request the necessary resources on demand. As long as there are enough resources avail-able these requests are met. This results in a system in which both continuous and abrupt resource demands can be met. Several reasons, including the lack of performance data per queue (consumers) or task (h-workers) and ill t scaling possibilities of the rst solution, make solution two stand out. It is subsequently developed into a proof of concept. Data retrieved from the proof of concept indicates a partial success. The queue system is in its current state unusable, mostly due to the lack of performance statistics per queue segment. It is therefore unknown what weight should be redistributed, making it a guess game. The manager/worker implementation works well for consumers. The setup is able to adjust its resource consumption to the demands while timely processing all incoming events. Due to facilities provided by the batch system, the manager/worker paradigm is robust and quite resistant to failure. The h-workers have not been tested, but provided there are enough resources available in the batch system a manager can request the required resources and receive them after a small delay. |
| Item Type: | Essay (Master) |
| Faculty: | EEMCS: Electrical Engineering, Mathematics and Computer Science |
| Subject: | 54 computer science |
| Programme: | Embedded Systems MSc (60331) |
| Link to this item: | https://purl.utwente.nl/essays/59196 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page