Replication Notes
-----------------
See: http://mule1.dataone.org/ArchitectureDocs-current/design/UseCases/09_uc.html
 
Replication of objects should be queued at the time the event occurs (CREATE, UPDATE, DELETE), and also on a scheduled basisin case event-based replication misses an event.  However, this hinges on how frequently the CN becomes aware of an event.

• Will MNs notify a CN of an event?
  • MN can inform a CN, but not implemented yet
    
    
• Will CNs poll MNs for logged events only? If so, what frequency?
  • Runs in a crontab-like function using quartz, every 2 minutes currently
  • Frequency is configurable by a member node (see the <synchronization> element in the node class)
    
    

Replication Authorization: 
NOTE: all references to "token" below should really be "session"
B = receiving member node (where the content is to be copied to)
A = providing member node (where the content is currently located)

• How will replication be authorized? Node-level certificate/SSL handshake?
  • CN makes replication request
  • Receiving node needs to know where the it originates from
  • CN must request both ends of the replication request
  • Create a group with all of the Tier 4 member nodes
  • need to add MNReplication.allowReplicate() on the replication node, called by the source node
    • pass in pid for object, target Subject
    •  
    • XXX0. CN calls A.allowReplicate(token, subject, pid)
    •  
    • 1. CN calls B.replicate(CNtoken, A, pid)
    • 2. B calls A.getReplica(Btoken, pid)
      • 3. A calls CNReplication.isReplicationAuthorized(Atoken, subjectFromBtoken, pid, Permission.execute)
        • other possible names: isSubjectAuthorized(), isReplicationTarget(), isReplicationAuthorized()...
      • 4. if (yes) A returns data to B
      • 5. B calls CN.setReplicationStatus
      • add in 'REPLICATE' permission (actually just use Permission.execute)
  • Replication get() logging should be logged differently than a regular get()
  • Should evaluate the Subject from the certificate. To evaluate, ReplicationService needs to track Subjects for each ReplicationTask it queues.  In IRC, we decided Types.Node needs to be modified to add a Node.Subject field, which is the value used to compare to.
  • We'll need two Hazelcast replication structures, and will also rely on a shared Synchronization/Replication Object Lock:
    • hzReplicationTaskQueue (this will be polled)
    • hzPendingReplicationTaskMap (this contains replication tasks that have been initiated, but not completed, and will be the source for finding ReplicationTasks that match a given Node.Subject)
    • hzObjectLock (a shared structure used to lock pids during SystemMetadata updates)
  • Possibly pass in a key (random string/subject string) that repl nodes verify

 
Event-based replication for a given object:
 
On create() or update(), check the SystemMetadata's ReplicationPolicy for:

• numberOfReplicas < count(CN.listNodes())
• Nodes listed in preferred list exist

 
Rank the target replication nodes based on:

• SystemMetadata for the given object with it's:
  • ReplicationPolicy
    • blocked list
    • preferred list
    • required list (is this defined?)
    • number of replicas as a minimum
    • replication allowed
    • (also, how to handle replicas that are externally maintained; set replication allowed to false, and min replicas to 0)
    •  
    • Decision: use only preferred and blocked lists; try to put a replica on all preferred nodes, plus any additional nodes needed to hit the min replicas
      • preferred > minimum : hit all preferred nodes, even if greater than minimum.
      • preferred < minimum : meet preferred and fill with more
      • preferred == minimum : stop at preferred & minimum
      • preferred - down < minimum : meet all available preferred and fill with extra
  • Replica list (location, status)
• Node rejection of a replication event
  • Once rejected, always rejected? Depends on the exception

 

• Initiate replication from the top of the list until the minimum number of replicas is created
• Need an internal timeout after the replication is initiated for a given node
• Create a ReplicationFailed exception
  • messages:
    • Replication denied
    • Replication timed out
    • ...

 
Node-based polling:

• If a node drops, use it's object list to evaluate whether or not each object's replication policy is fulfilled.  If not, queue the object for replication
• This requires a short list of objects on a given node.  This may require a filter on the CN listObjects() method.  We would need to pass in a NodeReference to get a list of objects on the given node. Question: Can we add a filter parameter to CNRead.listObjects() that filters by node id so we can produce that list efficiently?
  • Query solr index via mercury
  • Query metacat systemmetadata table

 
Question about ReplicationPolicy implementation:

• Should we replicate to numberOfReplicas only, or numberOfReplicas + 2 (or 3)? But not more? - Rationale: since the more replicas there are, if a node drops, there is significant overhead on the CNs to update all of the references to the replica node
  • initially stick with numberOfReplicas
• How long is too long for a node being down to initiate replication of all of the objects that were stored on that node? One week of downtime?
  • We need a notification step (email to the node manager)
  • Downtime for maintenance, a few hours, a few days
  • resolve() need to be updated to consider the status of each node