For the past year or so we’ve been advising on a German project to develop some data documentation tools for folks working with employment data. As part of the project, last month we went to Budapest to conduct a training on DDI, the standard for data documentation. We worked with the developers at OPIT, who are creating a toolset called Rogatus that works with the DDI Lifecycle standard. Last year we provided an introductory workshop to the same developers, so this year we were able to dive deeper.
We had a few goals for the workshop:
- Learn what’s changed in DDI 3.2 over the past year
- Get in-depth with DDI content
- Since Colectica works with the same standard, make sure Colectica and Rogatus are interoperable
Bonus: Besides accomplishing these, a side effect of having four developers spend a full week with our eyes on the DDI 3.2 schemas is that we ended up performing a pretty thorough initial review of the draft standard. We were able to submit dozens of fix requests to the DDI Technical Committee, and I’m thrilled that the fixes have been incorporated into the schema.
Changes in DDI 3.2 over the past year
The OPIT team is targeting the upcoming DDI version 3.2 instead of the current 3.1 release. They knew going in that 3.2 was a moving target, but the improved developer-friendliness of the update makes it worthwhile. Since the schemas the developers were targeting were a year old, the first day of the workshop was focused on bringing everybody up to speed on the freshly-released draft 3.2 schemas. Briefly, what changed over the past year includes:
- All item types can now be referenced; there are no places where items must be included inline.
- All types now have consistently-structured Groups (e.g., VariableGroup, QuestionGroup, ConceptGroup)
- Documenting datasets is much easier
- Specifying data types for questions and variables no longer requires an extra level of indirection with “delineations”. What were those anyway?
- Describing missing values is simplified
- A few of the new elements were renamed, like DataElement –> RepresentedVariable
DDI content covered at the workshop
It was nice to see the OPIT developers had DDI import and export functionality for a few of the element types already complete. The next step was to get into the schemas and look at more of the content, implementing more import and export functionality in a pair programming environment. The OPIT developers were able to add serialization for several new areas:
- Questions with multiple response types
- Question grids
- Attaching “other, specify”-style responses to code lists
- Question blocks, which are a group of questions preceded by some stimulus, like a picture or an article
- Survey instruments, including with conditional branches
- DublinCore citation information
All that was successfully validated against the new DDI 3.2 XML schemas, too.
Of course a week isn’t enough time to implement everything, but we also covered other material in enough detail that the OPIT developers won’t have trouble writing the serialization for these areas:
- Parameters and bindings in survey instruments
- Question groups (and since they’re consistent, all other types of groups)
- New data types, including nominal, scale, ranking, and distribution
- Datasets, and describing the variables contained in a dataset
- Storing summary statistics with a dataset description
- Studies and series
- Looping in survey instruments
- Packaging and organizing items (DDIInstance, ResourcePackage, Schemes, Groups, Fragments)
Yes, that sure is a lot of metadata. It was an intense and productive week.
Interoperability with Colectica
The official release of Colectica works with DDI 3.1, but we have started working on 3.2 support. As we worked on the Rogatus 3.2 serialization, we were able to test and make sure the XML output could be read into Colectica. Happily, it worked without a hitch. Things worked well in the opposite direction, too. It’s great to see two totally separate tools talk to each other, using DDI as the shared language.
Next week we are off to Copenhagen for another intense, weeklong Colectica and DDI workshop.
Get in touch if you are interested in hosting a DDI workshop of your own. Or if you would like to come to Minneapolis, we are always happy to have you here.