#persist rst
Notes for week of 2011-Sprint 45 (07-Nov - 12-Nov)
==================================================
:Backlogs: https://redmine.dataone.org/rb/master_backlogs/d1
.. contents::
Monday
------
Rob
- updated d1_common_java, propagated through MNode. CNode remains.
- made a perl script to create template code to assist process of ensuring consistency between api definitions and implementation. d1_common_java -> client code templates, and cross references to architecture docs
- Going to do the same for d1_libclient_java CNode today:
Notes:
did not update rmi interfaces yet, want to wait for cn implementers to be ready
(need to remove a bunch of InvalidRequests, and a couple NotFounds)
found 5 unimplemented methods (client throws NotImplemented exception):
reserveIdentifier
generateIdentifier
hasReservation
isGroup
isPublic
(should I implement them?, would like to)
Roger
- finished systemmetdatachanged call support
- django management command to process the list of updates
- updated CLI, and describe() to match the docs.
- working with cmd, cmd2 as alternative to simple command line
- Fixed the broken tests in d1_common.
- Working on converting CLI to using cmd2.
.. Note::
See: https://redmine.dataone.org/issues/2008
need a consistent way to define the root of each environment. Given the Root URL of a DataONE environment, a client can discover all it needs to know to interact with services available there. e.g. For the sandbox environment, a client may connect to "cn-sandbox.dataone.org" which should point to one of the sandbox CN implementations. From that, the client could pull the node registry and discover nodes and services available in that environment.
suggest::
Production: cn.dataone.org (RR DNS pointing to one of cn-ucsb-1, cn-unm-1, or cn-orc-1)
Staging: cn-staging.dataone.org
Sandbox: cn-sandbox.dataone.org
Development: cn-dev.dataone.org
Client instances should take a CN address as a parameter, this would be something that resolves to a CN baseURL, which could be a direct link to a CN, or an indirect link via the RR DNS mechanism.
Chris
- deploying cn-dev-2
- issue with replication manager not seeing the system metdata even after synchronization
- this may have just been a configuration issue. Uncomment the hazelcast.xml file location
on every deployment of the CN in metacat.properties. Looks like it may not have connected
to the cluster correctly because of this.
- mnreplication integration test needs to be updated to match the revised implementation
- Need to bring Metacat up to date with recent changes
- will work with current libclient impl, need to update again after cn signatures updated in libclient
DONE. TODO: Need to add NotFound exception to MN.getReplica() in docs
Ben
- replication bugs in metacat
Skye
- dev env sorted out
- working on list for web test apps: https://redmine.dataone.org/issues/1949
backlog:
Move https://redmine.dataone.org/issues/819 to another story, 907 to backlog
TODO: Need to implement the method restrictions as described in the Node List
TODO: need to keep cn-dev and cn-dev-2 up to date with each other (when necessary)
TODO: Test the issue with change system metadata even on hazelcast.
Tuesday
-------
- status of synchronization on cn-dev, cn-dev-2
- packaging of cn components
- status of metacat as production MN
Rob
- refactoring cn apis
Notes::
did not update rmi interfaces yet, want to wait for cn implementers to be ready
(need to remove a bunch of InvalidRequests, and a couple NotFounds)
found 5 unimplemented methods (client throws NotImplemented exception):
reserveIdentifier
generateIdentifier
hasReservation
isGroup
isPublic
- may be an issue with fewer cnode tests available for libclient
- remove RMI methods. Done.
Robert
- helping out Chris w/replication problems- isNodeAuthorized.
- committing code and updating old stories/tasks
- confirming synchronization still works on cn-dev/cn-dev-2
- rebuild cn-dev/cn-dev2 with ldap changes and jar/war changes
Roger
- fixed broken tests in common_python
- working through CLI
- Updating new CLI to use cmd2
Chris
- updateSystemMetadata on MNStorage, switch to systemMetadataChanged functions
- DONE. isNodeAuthorized not being proxied on the CN
Ben
- Bring metacat up to date following the libclient cn changes
Nick
- Met with ORC folks on hardware installation
- Waiting on 4 port NICS, VMWare licenses
- Need dedicated ports to enable console access to manage VMs
Wednesday
---------
Chris
- troubleshooting replication
- isNodeAuth working ok
- problems remain with serializing replication status, perhaps related to version mismatch on client lib
- taking about 20min to do a round trip (using cn-dev-2)
Ben
- Metacat upto 1.0.1
- Need to revisit how to register nodes, e.g. Metacat registration dialog
- how to generate certificates for a MN?
- Look into adding an xsl processing statement for XSLT
Roger
- Redid systemMetadataChanged - moved locking capabilities from management to GMN
- Continuing with CLI / CMD
Nick
- Working with VMs, need a Windows OS to access vcenter
- Can start setting up VMs
- Will check on speeding up processing for replication testing
Rob
- Checked in 1.0.1 common and client
- more refactoring to be done
- manifest metadata for versioning
- Updating webtester
Matt
- libclient caching implemented
- LRU implementation
- continuing with data packaging on client side
- next will be package support in R client
Discussion: Node registration.
- Metacat: need a button to start registration process instead of each time.
- More general issues for MN registration:
- MN ID generation
- Client certificate for MNs
- MN ID = 4 char, random, readable string.
- Certificates must be signed by a DataONE recognized CA
Process for registering a node:
1. Decide on subject
2. Generate certificate (with recognized CA)
3. Generate node document with subject and fake / null node id
4. a. Call register node (with a fake / null node id and the same subject as in the Node doc) to get real node id
b. CN checks that node is real and services are operating(a node may be registered before it is complete or even started. registration may occur by an administrator first submitting a node to reserve the node id and then working on bringing it up. registration will not be approved until the node can demonstrate its capabilities)
c. CN sets node id in MN node document
d. CN sets the verified status to false
e. MN records the returned node ID in its node document
5. Manual: Approve node reistration in node registry
a. approval process checks that node is real and services are operating
b. approval process sets the verified status to true
Action: Update metacat UI for registering the node
Action: Update the GMN implementation / mechanism for registering
Action: Design the administrative services and interfaces for managing Coordinating Nodes
Action: Decide on whether the "node verified" status should be a property in the node document. (It is.)
Action: ensure that the node update call is made by the same subject in the node document. (Nodes can and should be able to be updated by verified administrators of the system.)
Action: ensure that the CN.register() verifies that the incoming client cert matches the subject in the Node document, and that the Subject has not been used for other Nodes
Thursday
---------
Nick
fixing config issues in replication
hung up on setReplicationStatus() wrt serialVersion - getting an exception
Chris
working out issues in replication, with Nick.
might be an issue with creating rightsHolder subject when creating sysmeta
Subject strings question::
<submitter>CN=Christopher Jones A583,O=Google,C=US,DC=cilogon,DC=org</submitter>
<rightsHolder>DC=org,DC=cilogon,C=US,O=Google,CN=Christopher Jones A583</rightsHolder>
and::
<subject>cn=DEMO2,dc=dataone,dc=org</subject>
whereas DEMO2.pem contains::
/CN=DEMO2/DC=dataone/DC=org
How do we ensure Subjects are "equal"?
- Ben will look into CertManager method for normalizing DNs
http://static.springsource.org/spring-ldap/docs/1.2.0/api/spring-ldap/org/springframework/ldap/core/DistinguishedName.html#compareTo(java.lang.Object)
Ben
today - work on standardized subject
yesterday - added type marshaller to insert optional stylesheet reference into the xml objects
(NodeList, ObjectList, FormatList?)
Skye
- application list for Web Tester (index.html), still part of d1_integration, so is creating
separate project for it.
Rob
- question regarding difference between InvalidCredentials and Invalidtoken
- services currently ignore these because apache / tomcat throws SSL exception before the call reaches the
servlet / controller.
- what's the proper behavior of client when an SSL connection error occurs - should it throw exception, or proceed as public? (currently libclient_java continues as public).
Friday
------
Ben
- did the XSLT styles, wants to test on cn-dev on next deploymen
- did the metacat registration of MNs; not clear when to register versus update
Rob
- touching up MNWebTester- new version up there for 1.0..1
Skye
- split off web test suite into new maven project -- wants to talk to Dave before committing
-- having trouble with classloader issues
- not sure what's next in the pipeline -- will try to contact Dave
Roger
- task 1796 -- validation of checksum
- documentation for CLI
Matt
- working on EZID implementation library to wrap their web services for use on KNB MN
-- asked where to put it; agreement to add it to libclient