Notes for Standup, 2013-16-Block-2-4
====================================
Chris B.
--------
- 20130426
- Laptop upgrade troubles
- Planning on Ansible demo for Monday
- 20130424
- Ansible is now reading the config template file
- working on user input now
- Ansible walk through on Monday
- 20130422
- debian package work for Ansible-based packaging
- handling test cases
- shooting for finishing up this week
- Coordinate with Roger at UT next week
- 20130419
- Progress on the Ansible modules
- writing deb-conf - interacts with debconf backend
- writing 'sitefacts' module
- Uses the XML config file to build a list of the variables for the environment that module is run in (configuration info)
- plan to finish this up next
- 20130417
- Working on XML Schema for configuration under Ansible
- 20130415
- Meeting with Robert: Ansible
- agreed on XML schema for configuring CNs
- values from debconf
- user input
- template values, etc.
- modifying the debconf
Rob
---
- 20130426
- Finished up libclient ORE validation work (according to new documentation on this)
- Working with RDFS reasoners in lib_client_java - inverse relationship inferencing
- 20130424
- Working on loose end tickets in Block 2 wrt mutability, d1-client, etc
- Worked on resource map validation in d1_client_java
- fails on problems, needs to report better
- use a straight exception
- Dave: aggregation ID can be resource map URI + a fragment
- Dave and Matt will review the DataPackage documentation
- 20130419
- Working on isDescribedBy relationship in ORE documents, and how it's used
- working on integration tests for checking resource map construction
- Will add in story on validating recource maps on the CN (https://redmine.dataone.org/issues/3720)
- require both sides of inverse relationships as a D1 requirement, to be loosened later
- make it super simple to produce these maps by putting in libclient java and python
- create a service that uses RDFS reasoners to get the triples
- is there a way to limit all the triples that it processes? we only want describes, documents and their inverses, at least for indexing
- memory issues related to large models
- have the indexer make use of the library for obtaining the triples it needs to populate appropriate fields.
- 20130417
- Committed documentation for resource maps
- resource map parsing
- worked with Roger on python parsing of EDAC
- In Java: ResourceMap is a foresite object, we may not want others to have to code against foresite
- Meeting at UNM:
- content mutability discussion
- EDAC: content metdata + data into a zip file, and add another science metadata object
- Brings up package delivery issues: use BagIt instead?
- Implement services to get and load BagIt packages via the DataONE API
- 20130415
- Working on bug for system metadata without a submitter subject
- CN tester submits without a contact (submitter)
- Metacat should be checking this value (CN.create handler)
- Needs to throw InvalidSystemMetadata
- create cn.create() integration test that pulls the metadata to make sure everything is OK.
- Working with EDAC on tests
- looking at their serialized resource maps
- failing to deserialize with the Java ResourceMapFactory
- should we throw an exception if we get a null map?
- How does the foresite python library handle this?
- have not cross-compared java and python libraries wrt serialization/deserialization
Roger
-----
- 20130426
- time off
- Set up back up system locally
- ONEDrive Ticket: 3640 is done - couldn't reproduce it
- artifact of a decision on setting up filters, jumping directly to editing
- Found another bug: ONEDrive discrepancies between the index and what is available locally (getting access denied)
- ONEDrive should handle this
- The index shouldn't be showing these pids anyway (Skye will look)
- Mockup screenshots for the U/A meeting
- met yesterday with Dave, Robert, others going to the meeting
- 20130422
- Added in libclient examples for Solr queries
- More mockups for ONEDrive
- zip file objects
- each object could be exposed as multiple formats
- plugin-based transformations
- meeting tomorrow
- Working on backlog of tickets
- 20130419
- Added ore parsing example for Rob
- parsing of example from Rob worked in python
- Added another ONEDrive mockup
- really need the 'displayName' attribute to make files look right
- 20130417
- Worked with Rob on ORE maps
- Rob will try to parse via Java, need to do vice versa with python
- Worked on Mockups
- added two more - geographical area - by country or by lat/lon - needs UI work for degrees minutes seconds
- writing up an overview of the mockups
- call with Mike Frame next week on usability meeting
- 20130415
- ONEDrive mockups demo on Friday
- Looked through mutability document: meeting?
Matt
----
- 20130426
- 20130424
- 20130422
- 20130419
- 20130417
- 20130415
- Need to ccordinate with SEAD folks
Skye
----
- 20130426
- Audit reporting work + unit testing
- Fixed the issue Matt reported re: CN host names
- 20130424
- Tracked down UTF-8 issues in ONEMercury
- Search and facet encodings had to be fixed to not use ISO 8859-1
- Will get this into the buildout
- Met with chris and Ranjeet for DAAC and USGS issues
- application/octet-stream
- will change format ids on the CNs
- removed content: Ranjeet will archive those
- USGS problems too - will update archived flag for old objects too
- Synchronization report would be helpful
- Replication auditing now
- 20130422
- ORNL DAAC resource maps issue
- can't confirm 1029 - MN has 1011
- mistyped resource maps, then fixed them?
- Chris will look into the Dave/Ranjeet email thread
- ONEMercury encodings
- EML XSLT views had ISO 8859-1 encoding declarations
- fixed these
- Will be at tech conference on Lucene in San Diego all next week
- 20130419
- Worked on UTF-8 encoding in ONEMercury
- tried this in dev, set the Tomcat Connector encoding attribute
- Looked at how the maven shade plugin works
- index task processing and task generator daemons, no longer being put into maven
- done in the buildout
- Need a bit of help on the missing objects on the CNstated in the object path map
- Back into auditing - XML output for the reports, with XSLT
- 20130417
- Re-indexed objects in stage
- Working through ONEMercury UTF-8 support
- Tomcat connector isn't config'd to UTF-8: need to change this in cn-buildout
- Production index counts:
- Some documents not getting copied to UNM (200 ish objects)
- some pids don't have the object path in hzObjectPath map (40ish pids in UCSB and ORC)
- Worked with Jing on indexing jar - made changes to POM for dependency-free jar
- create separate jars: one for shaded daemon jar, one for a jar without the shaded dependencies
- 20130415
- SEAD: bug fixes would need to go into production if we move SEAD there
- Working on Auditing reporting
- Updated ReplicationManager to use not only authMN copies of replicas a source
Ben
---
- 20130426
- Metacat work
- SANPARKS discussion with Judith Kruger - more content in their MN
- 20120424
- Will upgrade the KNB this evening - 2.0.6
- Dealing with backups
- Will be able to turn on log aggregation (fixed OOM)
- Upgraded the SANPARKS node
- PISCO is at 2.0.5
- 20130422
- Worked on notes since Friday for relationship handlings and RDF reasoners
- 20130422
- Testing and tagging Metacat 2.0.6
- Will fix hudson config
- 20130419
- restart of UCSB in production. postgres connection pool issue, portal was failing
- need to look into connection pool leaks
- Working with TFRI MN - ensuring CNs handle UTF8 character set
- 20130417
- Semantics meeting
- /portal vs /portal/ - do both work?
- 20130415
- TFRI is sync'ing in stage
- indexing may be out of sync
- Display of chinese characters needs work
- Title should have a translated version?
Robert
------
- 20130426
- Time off Thursday
- Testing on Wed with Chris B. wrt CN buildout packages
- Config files aren't available during the config phase
- will meet with Chris again
- 20130424
- debian package work - user input and overriding issues
- 20130422
- Time off Thu/Fri
- Working on debian packaging
- 20130417
- cn-buildout coding for using new XML schema for describing configuration
- prepping for usability meeting
- 20130415
- More meetings on Friday: spoke with Chris B.
- might have some more changes today
- multiple flags will indicate the source of a config param's value
- CNs look to be in sync after TC, slapd restart
- indexing is off in terms of counts
- will work with Skye on re-indexing the remaining
Dave
----
- 20130426
- redmine ticket review
- quarterly report
- travel and meeting preps
- outlining a number of new use cases related to changing properties on sysmeta
- need to spec out additional use cases wrt system metadata mutability
- how does one change the auth MN?, etc.
- 20130424
- ORE work - working on a parser using RDFquery
- many malformed resource maps in the dev environments. Will check into this. ONEShare?
- Working on dataPackages doc
- 20130422
- Traveling
- Javascript parsing of resource maps
- will be able to introspect ores in the browser
- software tools for querying Solr
- Maintenance of redmine, etc.
- Adding in sprints through remainder of project
- need backlog triage
- 20130417
- European Union group call - possible interactions
- Writing documentation
- General architecture introductions
- tool that encodes Solr query terms
- Patrice is interested in certain queries
- CCIT call on Monday
Chris
-----
- 20130424
- DAAC/USGS work with Ranjeet and Skye
- d1noderegister script
- adding registration info to MN checklist
- TODO: email Bob Cook re: DAAC
- Followup: mutability discussion
- Next steps: work on detailed impl (API calls, handling sids, etc)
- 20130419
- TODO: ticket for postgresql connection pool issues
- SEAD: new object formats in Stage, working with Kavitha on rightsHolder entries
- Metacat replication troubleshooting
- Meetings with Matt, Jing on Metacat
- Meeting with LTER sensors group
- 20130417
- Rob: generated the EDAC test cert, access set to sscott, rnahf
- Working on Metacat locally - blocked by some configuration issues
- need some time with Ben for some configuration troubleshooting - after standup?
- SEAD: In touch with Kavitha, but not hearing back. Assuming the stage env will be used for screenshots for their NSF review
- Sensor network best practices writeup work
- Chris B./ Skye: coordinate on d1_indexing work
- Mutability discussion email out to Dave - not sure if he's on leave at the moment
- Robert: okay to add cn-orc-1 back into the RR?
- Chris B: IP address for splunk.test.dataone.org? In redmine ticket
- TODO: Ticket for implementing services to get and load BagIt packages via the DataONE API
- 20130415
- SEAD update of system metadata
- Metacat: troubleshooting replication update calls
- Sensor network workshop writeup
- TODO: schedule mutability discussion
- TODO: internationalization discussion (CCIT?):
- should DataONE require minimal english metadata fields?