Home Integration Demo Details Architecture Project

2. Details

2.1 Duties

Duties are the smaller processing unit. It’s a generic entity, wrapping anything transportable with significance only to the client. It might mean a signal for a process to start, input data for a function, a task name, a mesage, or simply data to be requested, uploaded, saved, analyzed, etc.
In the current version, Minka loads them once and keeps them always assigned to shards.

2.1.1 Creation attributes

name type note
id string Max: 256 length, a unique identifier across the application instances.
pallet id string Max: 256 length, required to link both entities inside the minka domain.
payload bytes optional, serializable object representing input data or cargo for a function or process.
weight double the cost of performance of processing this duty at a shard. Must be in the same measure unit as the weight reported for the pallet at the shards. It may or not be related with the payload

Considering the life-cycle of a duty there are two types of duties: Idempotent or Stationary. (not supported in current version)
For proper desired management, this’s a concern to tell minka about your duties. Because Minka can attach or dettach them to/from shards in any moment, as there’re many reasons subject but not limited to:

  1. cluster health states variation
  2. different new duty weights creates unbalanced cluster nodes,
  3. your application instances may be manually stopped or started,
  4. you add new shards and delete others,
  5. network partitioning may occur,
  6. unexpected hardware or software failure, making the shard fall

All this conditions translate to duties being re-located, without impact if they represent static information having few dynamic impact. But if they represent some sort of input for a long-term job composed of ordered tasks, then your processes must be prepared for savepoints. This’s due to its distributed resilient behaviour, and the multi-purpose nature of Minka, i.e. what you do with it.

2.1.2 Weights

Three things to note about duty weights:

  1. Duties have a weight, so the client can achieve a significant distribution order in their own meaning.

    • The duty weight is set at duty creation time by the Delegate. It must be calculated once and then change only in case of need. If the weight changes: a rebalance may occurr, and the duty may migrate among shards.

  2. Shards report their weighing capacities to the Leader

    • For this is required the Delegate to call minka.setCapacity(Pallet, double) in order for Minka to know in forehand how are shard’s enabled to receive pallets and duties, before balancing phase, which will fail to run if there’re shards not reporting their capacities. This double value should represent total capacity regardless of the weight already distributed by Minka. Know that this’s subject to change, the effect may make Minka to reduce or increase duty load on the specific changing shard.

  3. Weights are used by the Balancers to measure a pallet’s weight at a certain shard, determining the result of the algorithm, see the Balancer’s section below.

For example:

if a duty represents… the weight might be measured and shards reported capacity might reflect…
a file to save in harddisk in bytes the total harddisk space for that pallet
a file to upload at discretized locations in a coefficient of bytes and network latency the network uplink bandwidth with a coefficient of location in the network, that we want for that pallet
an image to analyze in a coefficient between bytes and file format might reflect a derivative of cpu clock speed and threads, that we want to assign to that pallet

It’s recommented that the duty weight value to be estimated: be more safe than accurate, and free of any factor subject to realtime variation, to avoid continuous redistribution by the balancers, causing a counter-productive effect on short-term duties.

2.1.3 Duty life-cycle

The lifecycle of a duty starts at creation and ends at removal. For both client requests to end up as attaching/dettaching events in a shard, among other events that also requires consistency management: there’s a state flow that Minka follows:

name type desc
create, remove client action request sent thru MinkaClient methods that triggers minka events
rebalance minka event occurrs when the Migrate function effectively changes distribution
attach, dettach minka event translates to follower events: capture and release, caused by create, remove and rebalance
prepared duty state developed at balance phase, before being sent
sent duty state after the duty operation is transported to the follower, in pending state
confirmed duty state after the previous operation transport is confirmed by the follower
dangling duty state lost duty detected after dissapparition of a shard. Causes a rebalance.
missing duty state detected difference between partition table and a follower’s heartbeat. Causes a rebalance.

2.1.4 Pallets

A pallet is a 2nd-level order into which we can granularize our processing needs. Pallets can be used to group together similar duties, that we might need to react with the same function() , or with the same pallet’s payload, or at the same pallet’s shard (in case we use the appropiate balancers for this).
Like with duties, Minka ignores what pallets are used for, but applies the following rules:

2.1.5 Creation atributes

name type note
id string Max: 256 length, a unique identifier across the application instances.
payload bytes optional, serializable object representing pallet’s constant data or cargo for a function or process
balancer metadata strategy one of 4 currently provided balancers to run on every balancing phase, to distribute duties on shards, with optional execution parameters.

2.2 Balancers

Balancers move duties from a source shard to a target shard with a functional criteria, by means of migration or creation. They have a fixed behaviour and continuously perform the same task no matter what cycle are in.

Reference for the following pictures


There’re four balancers that can be used for each individual pallet, at creation time thru:

PalletBuilder<String>.builder("pal").with(Balancer.Strategy.FAIR_WEIGHT).build()
Unweighted Weighted
Even size (balanced)

function in the round-robin style, using a circular collection of shards, keeping an even number of duties among the shards. Useful when duties represent other than resource exhaustion



Fair weight (balanced)

function that assigns -in terms of duty weight- a heavier load to bigger reported capacity shards, and lighter load to lesser ones. Useful for elasticity or variable shard sizes.

Spill over (unbalanced)

function that fills a shard to its limits, before going to the next shard, and so on. Useful to incrementally grow usage of shards as they’re really needed. (note locked pallets)



Even weight (balanced)

function that keeps same weight load on the same pallets at different shards. Useful for duties representing coordination tasks or business logics. (note identical sizes)

Sample of an application using all available balancers on different pallets

pallet size balancer note…
Green 26 fair weight it can fit only in two shards. One shard responded zero allocation for this pallet.
Red 15 even weight it’s present in all shards with the same weight.
Blue 13 spillover It filled the 1st shard now is coming for the 3rd.

Purple 6 even size It’s present in all shards with the same number of duties.

Then there’s a White synthetic duty (id’ed 55), it’s only one but it’s present in all shards.

2.3 Shards

Minka takes common known lessons on clustering like data replication, dynamically interchanging leader and follower roles, data sharding, asynchronous communication between nodes, etc. The following is an interaction diagram driven by its main actors.

They represent the union of the host application and the Minka library, which encompases these major ideas:

The leader election relies on Zookeeper, thru apache curator framework. When a leader fails, other ISR shard becomes the new leader.

2.3 Communication

Minka has a TCP socket event broker to asynchronously interchange messages between shards, built with Netty.
Although brokers are directly connected, they dont talk, i.e., broker clients don’t wait for an answer and broker servers don’t produce it. They both serve a contract, staying functionally asynchronous for fluid though slow-paced orchestration, leveraging damage contention.
Only a TCP port is needed to which brokers connect each other.

Resiliency

Cluster member restart
At the event of a clean restart, that is, the host application’s JVM ordered shutdown.
If the shard has leadership on the cluster, any other member of the partition’s table in-sync-replica will won leadership.
Duties held by the temporarily restarted member will be released before shutdown, and the leader will reattach them at other shards. This depends on the balancing rules of the pallets holding the duties.

Cluster member partitioned
At the event of a split of the ensemble, those still connected to the leader will keep working and
those unseen by the leader will release their duties waiting for the leader’s clearance.
The leader will left those shards at the shadow and redistribute their duties into the members still inside the leader’s subnet.
In case the leader is left alone at a subnet while all others still see each other, and all network partitions are still connected to a working zookeeper ensemble: the leader will consider itself as ghost and will decline leadership, provoking the followers still connected to zookeeper to reelect a new leader as usual.
In case the cluster is fully splitted, that is no shard has connection to others, no leader will trust stability.

SPOF
Minka has a single point of failure related to its dependency using Zookeeper (ZK) for leader election.
At the event of leadership election, that is the first time the cluster boots up, or when the leader have just went off, and a reelection takes place.
ZK must be available or all shards will release their duties an no entity will be assigned.
When a leader exists, ZK can fail without affecting Minka. If ZK and the current Leader fail at the same time: the Spof applies.

Leader-Follower health rules
If a follower ceases to receive or flaps the clearance from the Leader, which is a proof of communication, it releases its duties.
If a leader ceases to receive heartbeats from followers, their duties are re-attached to other followers, and the follower set in quarantine, until its beahaviour allows it to get back to the ensemble and receive duties.
If a follower flaps its heartbeats to the leader or like before they’re unseen, health ranking goes down, and the shard is set to offline and the follower will lack of clearance, which will make the follower to release held duties.