Implement a peer-to-peer membership daemon to be used for certain state

Implement a peer-to-peer membership daemon to be used for certain state
synchronization in future distributed / decentralized IMUNES operation.
The daemon will try to connect to remote peer(s) specified as command-line
arguments at invocation time, and form an ad-hoc peer-to-peer overlay
network with all nodes reachable via its peers.

Each node in the peer-to-peer structure is uniquely identified by its
IPv4 address.  The daemon will try to maintan a small number of direct
peerings (between two and four) between random nodes in the overlay, thus
forming a well-connected mesh over time.  Each node maintans full routing
information to all other nodes, basically in the same way as BGP does,

Implement a peer-to-peer membership daemon to be used for certain state
synchronization in future distributed / decentralized IMUNES operation.
The daemon will try to connect to remote peer(s) specified as command-line
arguments at invocation time, and form an ad-hoc peer-to-peer overlay
network with all nodes reachable via its peers.

Each node in the peer-to-peer structure is uniquely identified by its
IPv4 address.  The daemon will try to maintan a small number of direct
peerings (between two and four) between random nodes in the overlay, thus
forming a well-connected mesh over time.  Each node maintans full routing
information to all other nodes, basically in the same way as BGP does,
except that instead of AS numbers we use node ID-s (IP addresses) to
construct path vectors.  Once the routing state converges, no topology
information needs to be exchanged, except periodic keepalives used to
verify that direct peerings are active.  Hence, in steady state the
protocol is unlikely to consume any measurable network bandwidth nor
CPU time.

Besides maintaining the topology / reachability state, the daemon provides
a simple facility for nodes to announce arbitrary attributes associated
with their IDs.  The attributes will be distributed by flooding the
overlay network with new state.  Only the attribute set with the version
number greater then the currently stored one will be propagated through
the overlay, thus preventing endless loopings.  Hence, the originating
node is responsible to bump its attribs version number each time it
attempts to broadcats a new set of attributes.  In the future this part
of the protocol might need to be enhanced so that only incremental /
partial updates would need to be sent.

An application can directly interface with this "daemon" by observing
global variables "active_hosts" and "dead_hosts" which will be updated
dynamically.  For each active host the host_attrib_tbl($host_id) should
store most recent attributes, if any.  If the need arises, notification
hooks can / should be placed in ProcessAnnounce, ProcessWithraw and
ProcessAttributes procedures.

The framework was tested on our ad-hoc cluster with 1032 virtual nodes
mapped to 8 physical Pentium-4 machines.  After a relatively long initial
synchronization period (around 20 minutes, mostly CPU-bound) joins
and leaves to the overlay are processed and propagated to all members
virtually instantenously.  However, in sporadic cases topology changes
can lead to shorter periods of oscillations lasting up to 10 - 20
seconds, but those oscilations are typically observable only on a
limited set of nodes.

My initial impression is that the protocol should work fine for overlays
of up to several hundreds of nodes in size, at which point we should
investigate alternative options for maintaing the overlay coherence.

Bug found by:
Submitted by:
Requested by:
Reviewed by:
Approved by:
Obtained from: