CCIT meeting 2010-05-25,26,27  Minutes
------------------------------------------------

Vieglais, Jones, Wilson, Palanisamy, Berkley, Horsbourgh, Scherle, Waltz, Allen, Cobb, Chris Jenkins (Boulder), Dahl, Servilla, Nahf, Sandusky, Lapp


Around the room
-----------------------
Introductions from all CCIT participants
Jenkins -- dbSEABED -- data framework for seafloor soils data 
http://instaar.colorado.edu/~jenkinsc/dbseabed/
        

Parking Lot
-----------------
See http://epad.dataone.org/20100525-ci-parking-lot

Review of Prototype status
------------------------------------
Current components:
    - Design docs
    - 3 CNs
    - 3 MNs
      -- Dryad MN -- uses Python stack
      -- DAAC MN -- uses Python stack
      -- Metacat MN -- implemented directly in Metacat
      
    - web UI 
    - ITK libraries
    
    See overview diagrams: https://repository.dataone.org/documents/Projects/cicore/architecture/components.pdf
    
Implementation matrix for Service APIs
----------------------------------------------------

Method                                      |  Python GMN   | Metacat MN
-----------------------------------------    ------------------    -----------------
MN.get()                                    | x                       | x
MN.getSystemMetadata()         | x                       | x
MN.create()                               | x                       | x
MN.listObjects()                        | x                       | x
MN.describe()                           | x                       | x
MN.getLog()                              | x                       | x
MN.create()                               | x                       | x
MN.update()                              | x                       | x

Coordinating Node overview
  -- Object store, search index, search are working
  -- metadata packager and parser are functional
  -- harvester ?
  -- replicate queue & replicator are to be built
  -- working to refactor to use Spring batch processing to streamline number of JVms running and give a standard processing framework
  -- Do we need to include MN replication?  Probably not required, but good if we can
  -- Issue: harvester isn't the only place where objects get inserted into CN Object store
     -- need to be sure to queue indexing and replication after CN-CN replication
     
Reviewed the overall Version/API matrix: http://mule1.dataone.org/ArchitectureDocs/versions.html

Main tasks remaining are associated with:
  -- CN development
  -- Deployment on various virtual machines
  -- Integration testing
  -- further development of client libraries + ITK components
  -- need to plug the authentication/access control holes in current metacat MN implementation

listObjects() == search()
search() /object/
get()   /object/<guid>
getsysmeta()  /meta/<guid>

/resolve/<guid>

&&&&& Task: how is search syntax passed into REST interface

&&&&& Task (Chad): agree on and implement authn & authz for short term deployment

&&&&& Task: (Giri) determine how we can take advantage of the science metadata that Mercury can return now to enable clients to specify what fields they want to come back from the search and then return back that information in the search results.  

&&&&& Task: (Roger) update Python MN to match new REST API interface.  

Breakout groups
----------------------
CN stack completion
Integration testing

Member node implementation issues
Deployment issues

Security -- IP based only





CN Stack Completion

http://mule1.dataone.org/ArchitectureDocs/REST_interface.html#object

- Index updating.  The Mercury index needs to be kept up to date as content is added to the Metacat object store.  This requires triggering a work flow that involes:
  1. package system and science metadata when new science metadata is stored in metacat and store the package in a temporary location
  2. parse the package to extract the necessary values for populating the SOLR index and perform an incremental update to the index

Metacat logs create events, so a simple mechanism for triggering the index update workflow maybe to monitor log output from metacat.  The log should be modified to write out the identifier of the science metadata, and the local identifiers of the science metadata and the associated system metadata.  The process that reads the log messages and invokes the packager will need to track state so that it does not re-process existing entries (though technically duplication should not affect the indexing accuracy, just efficiency)

- Story: modify metacat log to contain necessary information - identifier, local ids

- MN Synchronization API 

  - Story #592 complete / review process description in the archtiecture docs (use case 06)
  - Story #595 build the harvester component that will retrieve content from a member node given an identifier and member node information
  - Story #608 Task #629  define the messages that are used to control harvester 
  - Story #608 implement the message queue
  - Story #610 define the mechanisms for populating the message queue
  - Story #609 initiate processing by cron signalling 
  - Story: Wrap the packager into a  synchronization job
        - on Major CN packaging is executed by signal from completion of Create
  - Story: On minor CN's disable most MN syncronization API excepting the packaging queue
  - Story: On minor CN's enable inotifier to signal execution of packaging job
  - Story: Modify Metacat to insert a message into the packaging queue whenevera a CRUD or replicate event has completed


- CN Replication API
  - mostly integration testing with perhaps some iteration

- Rest API
    (New Story) As the System Architect , I should define the mechanism outside of Mercury and Metacat in which CN tracking information should be persisted. (Mysql/Postgre/??)
    
    CN Query
        CN_query.getLogRecords() (review project backlog and match up those stories / tasks with entries here, and add new stories / tasks as necessary)
            Story: As a developer, I need to design a   LogRecord  schema
            Story: As a developer, I need to know what events should be returned in the logs (for metacat and mercury and cn stack)
             Story: As a developer, I need design integratation of Metacat and Mercury in order to implement getLogRecords()
            Story: As a developer, I need to implement the integration of Metacat, Mercury and CN proxy layer
             Story: As a developer, I need to maintain uniform time stamps from the CN system components
             Story: As a developer, I need to create a servlet that exposes the LogRecord result set to a query
        
    - Sequentially call MN.getLogRecords() with same parameters used to call CN.getLogRecords() and return the combined list of log records.

             
    CN CRUD
        Story:  As a developer, I need to modifey the rest proxy controllers and Rest Service Webapp to conform to the new REST interface specs
        CN_crud.resolve() -> list of identifiers
           Story: As a developer, I need to design and implement the resolve method.  This requires querying metacat for replication locations in the system metadata associated with an identifier.
          
        CN_crud.reserveIdentifier()  ** Not for Prototype **
           Story: As a developer, I need to design and implement the reserveIdentifier method. This includes determining where  state information about the reservation will be stored, and implementing  the method in cn-service.

- MN Replication
  probably not for July?

- Metrics recording / reporting
 -  Story: determine which metrics should be recorded (this will be a superset of the metrics that need to be reported to NSF).  Should include low level system information useful to administrators / operators. 
 

- Monitoring (nagios)
    Story: As a developer, I need to design a monitoring system that can be used centrally monitor and report on MN service availability, and monitor CN system and service metrics (disk space, cpu, etc.)  This includes providing deployment details, communication mechanisms across public network, security mechanisms, and notifications to system admins of MN and CN.
    Task: Define the metrics that need to be monitored and recorded
    Task: Select the monitoring framework and design the deployment of the selected framework, including gathering the necessary information for configuration
    Task: Deploy the monitoring system central monitoring service
    Task: Add the necessary packages to the CN deployment packages
    Task: Create a plugin for the MNs that can report on availability of the MN - this could probably be implemented as a HTTP method call on a MN, e.g. MN_health.ping()


    


Integration Testing
See http://epad.dataone.org/20100525-ci-integration-testing


Wednesday, May 26 -- Block 5
------------------------------------------

Planning for Year 1 Milestone

REST URI patterns: 
Discussion of the integration testing (see http://mule1.dataone.org/ArchitectureDocs/integration-testing.html for Matt's summary of the results of the discussion on Tuesday and http://epad.dataone.org/20100525-ci-integration-testing for the raw notes).  Not all of the high priority tests need to be done for July 31st, since some of them represent features that won't be implemented for the year 1 release, but the high priorities will need to be accomplished for the initial public release).  

Harvesting:  Metacat assumes that the harvester is provided URI's.  Need to populate a sitemap.xml with the URL's, which the harvester will go get.  Note that the get function is a REST URL.  May need to call the create() mechanism to make the appropriate objects.  ???Further discussion suggests that leveraging the harvester in Metacat or Mercury would be more work than creating a simple harvesting application.  Create method, when it completes, either writes out a log message or a JMS entry.  Sentiment seems to be that JMS is better, to guard against things like changing the log level to null.  JMS message "done creating, here's the GUID".  Problem with coordinating nodes replicating.  They create content in Metacat, which may not go through the create().  But JMS isn't in Metacat, at this point.  Metacat logs the initial CRUD operations, but doen't differentiate a replication event as separate.  

Story: Metacat, on completion of CRUD operation, will send a message to a package queue to indicate a change in content that needs to be processed.  Message needs to contain the two local identifiers plus the external identifier.  ??Will system metadata be treated differently from science metadata?? Chad is passing the system metadata identifier in the replication message.  

Story: The CN needs to maintain a list of registered Member Nodes and their service endpoints and minimal additional metadata that can be used by the harvester to iterate over Member Nodes to initiate MN synchronization

Story: Need to implement the mn.ping() as part of the health monitoring (may be part of nagios monitoring)

Story: Need to design and implement a log aggregation facility at the CN to be sure we have all logs (even if MN goes offline) and can respond to log requests without a distributed query

Story: Need to design and implement a statistical counter on the downloads for data sets and metadata objects that can aggregate at daily, weekly, monthly, and annual time scales.
- also for blocks 1&2 in the performance metrics spreadsheet using System Metadata at CNs.

 (Usage Statistics) &&&&Task: Cobb and Jenkins to provide suggestions about existing systems that handle distributed log aggregation and log statistical summarization

Story: Need to design and implement a system for accumulating uptime and other metrics fromt he nagios monitoring system for use in reporting at daily, weekly, monthly, and annual scales

Investigator Toolkit: will have client linbraries fleshed out for integration testing, so take advantage of these to implement reference ITK tools:
1. (highest priority, start with python client lib) command line clients based on client libraries
2. R add on for search and retrieval of content from D1 infrastructure.  Even implementing get() would be a valuable addition (given D1 identifier, retrieve data and store in R workspace).  (Note that it would be very beneficial to ensure local caching is used to avoid multiple retrieval of objects)
3. Investigate how much work would be required to implement in Kepler (search and retrieve)


Usage metrics
Examples (Multiple servers, Data serving, US gov/NSF):
Tools for Project + Task Management

- Agilo has many issues - performance, reporting capabilities
- Generally difficult to get a high level view of development status
- For next stages of project, attempt to fully specify WBS

- Features
  - easily divide tasks into sub-tasks
  - Code review functionality
  - efficient user interface (easy for devs to work with, fast)
  - High level overview for managers, reporting
  - Drill down from high level view to tasks remaining
  - Time estimation
  - integration with subversion
  - integration with development environment (e.g. eclipse)
  - integration with LDAP for account management
  - dependency management (between tasks)
  - comparing snapshots from differnet stages of sprints / between sprints
  - tracking bugs as part of release cycles and milestones

- Review task management tools
  - Version One
  - FogBugz
  - Jira
  - PivotalTracker
  - Redmine
  - Basecamp
  -Serena Agile (subscription model: $35/person/month, volume discounts)
  
  
  NESCENT review of tools at http://spreadsheets.google.com/pub?key=p6ikV0To6fXrzsm1de1W4Hg&output=html (from an E-mail by Ryan)
  
  (lists of tools at http://www.dmoz.org/Computers/Software/Configuration_Management/Bug_Tracking//  and http://en.wikipedia.org/wiki/Bug_tracking_system at the bottom)
- Try removing agilo plugin as first effort towards improving Trac performance.


Discussion: Year 2/18 months Goals and Priorities

Dave overviewed 5 focal areas for Year 2
Matt overviewed potential goals
    * Building production infrastructure
    * Focus on client side tools that enable data reuse and new types of science

Rob Nahf: metadata is key, could have focus on metadata production
    -- Chris Jenkins: have D1 operatives help with metadata collection eg via phone - this works !

Giri: also be able to enable advanced services (and multiple data representations)
    -- e.g., data subsetting (opendap, wms, modis subsetter, etc) and visualization services
    -- handling datasets that are in databases, rather than separate files.  
    -- handle metadata records that are not pointing to any specific dataset resource, e.g. metadata records that are pointing to a dataset directory, project pages, maps

Ryan: High level goal of starting on high-level problems we've identified
   -- streaming data
   -- data packaging
   -- Matt: also see semantics wg goals
   
Hilmar: willing to engage with leaders of WGs that are pending

Bob: could employ the CE model of bring in in leaders to draft the charters

Matt: no reason working groups can't be somewhat overlapping topically

Chris: Is there a place in the architecture for computational services that might be needed for advanced features (e.g., semantic indexing across whole D1)

Paul: EVA absolutely want ability to bring the computation to the data
  -- we should focus on things that will really benefit EVA and that type of activities
  -- Matt, Dave, and Paul will try to get list of issues from EVA in July

John: TeraGrid nodes could provide some of these computational services

Bob: Also could ask CE groups to help solicit these issues

Bruce: How do we utilize science metadata to describe services that are available for datasets, and deal with the different ways each metadata standard represent these?

Dave: How to deal with the transient nature of these services?  Can we help preserve services as well?  Could we monitor service availability?

Giri: Wind energy informatics based on GIS services; very successful with that community

Paul: What do OGC standards say about persistence, changes, etc?

Giri: There are some very reliable services too; national atlas; blast; maybe we could identify these so that they are much more obvious; can we catalog and monitor useful services?

Dave: leverage presistent IDs and ID resolution to help with automatically redirecting service requests

Paul: does this type of service redirection actually fall into the DataONE mission?

Matt: maybe only focus on the mainstream data subsetting services (WMS, WCS, opendap, etc) and facilitate locating those services

Bruce: could help clarify best practices of how to identify services in science metadata standards

Summary statements:
------------------------------
Bob: be sure to articulate with CE to have them provide direction
Rob: need for editing tools for metadata and data provision
Paul: starting work on the hard problems (data interoperability, semantics, dealing with web services, bringing computation to the data)
Robert: refactor the codebase to be production ready
Giri: handling metadata that are not directly pointing to a single data set (data packaging); handling the web services for continuous data streams or large data collections
John: Engaging new potential member nodes as friendly implementors
Chris: Data interoperability, it's the whole reason for doing this, as it enables widescale computing on the data (the working groups are key in this)
Jeff: avoid an identity crisis (people won't get it unless they have something they can use); must have impact on community practices sooner than later
Ryan: the world will be watching, so must have a production system that they need, so must connect with CE group to identify needs that have to be met
Hilmar: data findability will be critical, so semantic search and data integration should be emphasized; know our early adopters
Mark: authentication + authz; need to simplify and streamline it
Roger: deep provenance because its important for contributers to get credit when their data is used; we can facilitate that
Bruce: Robust production infrastructure that delivers useful data in a useful way to solve science problems
Matt:  Focussing on utility of the infrastructure to solve science problems in analysis in modeling; produces new science that would have otherwise not been possible or would have taken much longer.  
Dave: Making sure we have a robust system that bullet-proof within the timeframe and meets the anticipation of the community.

2010-05-27

Semantics discussion

- How to improve accuracy and precision of searches that can be conducted through the portal and also through the APIs

Helping with data integration (small steps)
- Resources for semantic analysis across the data collection include: Units of measurement, parameter names / labels, parameter roles in "text book" equations, manual annotations
- How much manual work is required / possible for annotating incoming data sets?
- Ontologies - prefer something that provides some emergent behaviour rather than relying on complete/tight definitions of existing ontologies
- Semantic routines are possible to indicate which parameters are "closer" to each other. This assists users in identifying which might be appropriate for a particular application (e.g. data aggregation, data visualization, model setup/validations)
- What is meant by an "ontology" these days (XML:RDF:SKOS:OWL) - range from simple lists/trees through to large graphs with different levels of expressiveness
- Building ontologies is a very time consuming, resource intensive process.  
- Need to take advantage of ontology development activities outside of data one, but relevant to project
- D1 needs to determine what use ontologies will have in the project

Search:
- support mapping variable names / parameters to higher level concepts.  e.g. search on "air temperature"
- searching/finding and interrelating datasets = will be key attractors for new data/users to D1

- How can these annotations be pushed back to the data sources / member nodes?  D1 needs to provide mechanisms that enable this.
- WG topic- how to manage the flow of annotation information throughout the infrastructure?  Ensure that annotations are appropriately associated with original data + metadata
- Jeff - HIS model may be appropriate for evaluation in D1
    The HIS model includes a domain ontology for variables that is centrally hosted.  data  publishers are responsible for ensuring their variables are mapped to the ontology
    http://hiscentral.cuahsi.org - maintains a data service registry, a metadata catalog, semantic annotations between variables and the ontology, and search services that operate on the central metadata catalog, which contains the semantic annotations.  Search services are at http://hiscentral.cuahsi.org/webservices/hiscentral.asmx.  Documentation is not great yet.
- also:  VSTO, SERONTO (data system - MORIS)

- How to integrate existing activities / products into D1 infrastructure without repeating the effort?

- SONET work is quite relevant - July meeting is a good starting point for more interaction with D1 - also DC will be at meeting

- Earth system grid also working on similar problems (note: ORNL is an ESG node; Giri is involved with this); heavy emphasis in FY11 on model-data integration.  Major DOE proposal being developed across labs.  
- http://www.vivoweb.org/ - a cross-university project aimed at connecting researcher identity, published work, research interests. Goal is to make identification of potential collaborators easier. This and other similar efforts may be path to linking research identity / published output to datasets in DataONE.

- Biomedical ontologies also relevant 
- PATO Phenotype and trait ontology
- Open Ontology Repository may be relevant
    -- MMI OOR repository has a particularly relevant collection of ontologies

Important for D1 to leverage existing activities - through services is one mechanism - how to make service interaction efficient and scalable? 
 e.g: - HIS web service: http://hiscentral.cuahsi.org/webservices/hiscentral.asmx
       -  NBII thesaurus web service http://nbii.ornl.gov/thesaurus/

NBII Clearinghouse is currently working on enabling thesaurus in search refinement (broaden or narrow the search results using the thesaurus keywords)

Streams:

- Search
- Integration
- Interactions with existing and emerging related activities
- Formulate proposals for expanded work on related topics
- Driving exploratory data visualization
- Storing annotations (e.g., the open annotation project; http://www.openannotation.org/theOpenAnnotationCollaboration.html)
- Data format registries - helping to  drive the types of processing 

Actions:
- Jeff and Bruce will draft an outline of topics for semantics groups as input for the July 19-20 (approx) Semtools meeting in Ithaca
- Merge notes from previous meetings to help identify major topics / streams
- Incorporate overview in the architecture docs


Block 10

Interactions between groups

- Try and attend other meetings - e.g. EVA
- Ensure the calendar of meetings (f2f and vid conf) is available to all members of the project
- Be opportunistic about attending meetings where possible - even if not for the whole meeting
- liasons for leading interactions


Dave - the working groups as noted on the DataONE website indicates that "education and outreach" are split and bound with "community engagement and education" and " citizen science and outreach", respectively; should the list below be refactored to reflect the working groups that are outlined on the website or have there been changes in the list of which I am not aware? - thanks, Mark Servilla

sociocultural (Bruce + Robert)
usability and assessment ( Bob Sandusky, Giri)
sustainability and governance (John)
education and outreach (Mark, Ryan)
EVA (Matt, Dave, Paul)
citizen science (Bruce, Paul)
LT + Sponsor (Dave)

Actions:
- Ensure calendar is kept up to date
- Suggest a weekly summary from all groups (CCIT + CE&O + ED)
- Issue register (for both CE and CI)

Meetings 2010 - 2011

- EVA       July 14-16 (Ithaca)
- SONET July 19-20 (Ithaca)
- CCIT     October 4-8, TDWG = 26-  (Woods Hole)
  - Matt may have a contact at MBL for assistance with logistics
  - Possible locations include NCEAS, Nescent, ABQ, Denver (USGS), Knoxville
- D1 AHM Nov 1-5?  Albuquerque, Tamaya
- Federated Identity Early Sept 2010 Sept 7,8,9 (two days)
- DUG Dec 9-10

Next CCIT meeting:
March, April, w5 May, June
(week of july 11 or 18: ESIP in ABQ)

Actions:
- Determine possibility of merging CCIT + AHM meeting for Nov 1-5
- Contact Randy and lock down dates for FS meeting (Mark will take lead on this)
- Doodle poll for next CCIT meeting in the March - June time frame)