new cloud member connects to any existing cloud member cloud members have: - cloud-wide master key, K - cloud IP cloud members attempt to establish a complete graph of connections delays get flooded throughout the cloud in an attempt to use well-working paths B(x) is a one-byte string whose byte contains x. HASH(s) is the SHA256 hash operation. EECB(p,k) is ECB Rijndael encryption of p with key k. p and k must each be 32 bytes long. DECB(p,k) is ECB Rijndael decryption of p with key k. p and k must each be 32 bytes long. CRC32(s) is CRC-32 with polynomial edb88320 and initial value 31bb29b0. on connection: - listener sends 16 random bytes, Rl. - Connecter computes Rc = 16 random bytes t = CRC32(K||Rl||K||Rc||K) b = one-byte XOR of all bytes of t xs = B(b) || B(b+1) || ... || B(b+31) s = Rl || t || Rc hk = HASH(K) for (i=0;i<64;i++) s = EECB(EECB(0,HASH(K||B(i)||s||B(i)||K)) XOR xs, hk) and sends Rc || b || s - Listener checks this. On failure, listener silently throws away all state, leaving the connecter with a half-open connection. On success, listener sleeps for a short time (a few seconds) before continuing. - Each end computes Kl2c = HASH( "->" || K || Rl || "->" || Rc || K || "->" ) Kc2l = HASH( "<-" || K || Rl || "<-" || Rc || K || "<-" ) - Kl2c and Kc2l are master keys in the listener-to-connector and connector-to-listener directions. In each direction, this key is called Kd; the other direction's Kd is Kd'. - Immediately after the above, each direction rekeys. To rekey, an end: - Generates 32 random bytes, Rk, and sends EECB(Rk,Kd). - Computes t = Kd || Rk || Kd' k = "" for (i=0;i<8;i++) t = EECB(HASH(t||Rk||B(i)||Rk'||t),Kd) k = k || t The resulting k is used as an arcfour key for data that end sends, with the first 64KB of the arcfour keystream discarded. The receiver must of course mimic the computation using the decoded Rk and the Kd and Kd' values the other end used, to obtain the key to decrypt the data stream. - Generates two random bytes, L1 and L2, and sends L1 and L2 as the next two bytes of data (ie, the first two bytes encrypted with the new arc4 keystream), L1 first. The number of bytes before the next rekey, ie, the number of bytes sent between L2 and the next rekey's EECB(Rk,Kd), is 0x3fff0000 + (L1 * 256) + L2. Senders and receivers must keep track of bytes sent and received to know where rekeys occur in the data stream. Data carried by the above low-level protocol consists of a stream of packets. A packet consists of length, type, and contents; the contents may be up to 65535 bytes long. The type can be one of IGNORE The whole packet is ignored by its receiver. PING PONG These allow a member to (a) determine connectivity to and (b) find out the round-trip time to another member. WHERE-ARE-YOU I-AM-AT These allow cloud members to learn where they might be able to connect directly to other could members. DISTANCE This indicates reachability of a cloud member. IP This is an IP packet for another cloud member. In detail: A packet consists of: 1 T Type 6d IGNORE 46 PING 1a PONG 91 WHERE-ARE-YOU 37 I-AM-AT e3 DISTANCE 8c IP (These were chosen randomly under the constraint that each pair of codes have a Hamming distance at least 4.) 2 L Length (big-endian) L Contents Detailed packet descriptions: IGNORE Contains anything. Ignored by its receiver. NAT-QUERY Contains nothing. Provokes a NAT-REPLY from its receiver. NAT-REPLY Sent only in response to a NAT-QUERY. Contains an IP address and port ID (see below), holding the address of the NAT-REPLY's receiver as seen by the NAT-REPLY's sender. PING Contains anything. Provokes a PONG from its receiver. PONG Sent only in response to a PING. Contains the same thing as the PING which provoked it. REACH Contains a list of pairs. addr is an IP address; delay is a delay in microseconds (unsigned, four bytes in network byte order). This gives reachability, from the point of view of the REACH's sender, of every cloud member the REACH's sender knows how to reach. IP Contains an IP address (see below), which must be a cloud member's address, followed by zero or more bytes of packet contents. This packet is routed to the cloud member whose IP it bears; that member then treats everything after the address as a reeceived IP packet. If the receiver believes the destination member is unreachable, it should silently drop the packet. An IP address is represented as a protocol ID byte followed by zero or more bytes depending on the protocol ID. When IP addresses are accompanied by port IDs, the port information is also protocol-specific and follows the address. Currently defined protocol IDs, with descriptions of their address and port formats: 0 IPv4 Address: four bytes in network byte order. Port: two bytes in network byte order. 1 IPv6 Address: sixteen bytes in network byte order. Port: two bytes in network byte order. Each member monitors the delay to each of its neighbours with PING/PONG packets. Each member also periodically sends out a REACH packet containing its local routing table; in the absence of topology changes,these are sent out approximately once a minute, with some randomness to avoid periodic traffic spikes: the time between each REACH and the next, in seconds, is a number uniformly distributed between 60 and 65. Given reason, such as a topology change, this time may be cut short and a REACH generated immediately. Each member maintains a list of other members it (believes it) can reach, with associated delays. Each node also, in the state for each neighbour, keeps the contents the last REACH received from that neighbour. If no REACHes are received advertising reachability to a member for five minutes, it is dropped from the local routing table (and from future REACH generation).