SIF Association

This blog is run by the staff of the SIF (Schools/Systems Interoperability Framework) Association for the use of it's members and the general educational community.

It is specifically designed for those who are interested in understanding what SIF is about, where the standard is going and how Schools, Districts and State Educational Authorities will benefit.

Friday, November 18, 2011

Migrating to a minor SIF release: Programming Conventions

In general the SIF standard issues a new minor release every 9-12 months, and that trend looks like it will be continuing.  These releases traditionally extend the SIF Data Model by adding new objects, new elements and new code list values to the previous version, although with SIF v2.5, an entirely new Web Service infrastructure stack was also introduced.  The following requirements apply to every new SIF 2.x minor release. 

·         1. Support Incremental Migration
It must be possible for a Zone Administrator to incrementally upgrade the Zone to a new minor release one component (ZIS, Agent / Application) at a time without:

o   Losing existing functionality or “breaking” anything
o   Encountering significant interoperability problems during the upgrade

The ordering of the component upgrades may be fixed by the release documentation (ex: ZIS upgraded first, then Object Providers, then ...)

·           2. Limit  “backwards compatibility” logic requirements
The early developers of a SIF agent or application conforming to the new release will have their initial deployments occur in Zones where majority of the pre-existing SIF agents and applications are only supporting the older versions of the standard.

It must be possible for the new agent / application pair to integrate into such a Zone relatively easily, without requiring the developers to incorporate “significant” internal logic to translate between the new and old versions of the specification.

The first of these requirements lowers the cost of migration for end users, while the second does the same for agent and application developers.

In order to meet these goals, it is necessary for legacy components to conform to the following long established “robust in the face of change” programming techniques

1. Accept and Ignore unexpected elements

In many approaches to XML versioning, the only truly “backward compatible” change is one in which the existing schema is somehow restricted (as when a formerly optional element becomes mandatory).  This type of change ensures that any new message will validate using the old schema since effectively the optional element in question just “happens to be present” (because it is now mandatory) in any message sent by an application conforming to the new release.

On the other hand, consider the problem when a new optional element is added to an object in the new release.  It may be included in either an Event or Response message sent out by a newly installed application. While the legacy recipient can examine the “Version” attribute on the message, it can’t select a corresponding XML schema to use in validation when it was written in conformance with an older version of SIF.  If XML validation is turned on, the incoming message will not validate using any schema the recipient understands.

The problem here is that one of the most natural ways a standard develops is by adding new and previously undefined elements, and the SIF standard is no exception.  Assuming applications supporting these new elements must be integrated into a Zone with older components (which initially is ALWAYS true), how can this be done without either updating all affected applications simultaneously (a non-starter) or breaking existing functionality (unacceptable)?

The answer is rather obvious. Agent developers can use an XML parsing approach that simply ignores any element which is unexpected or not defined in their schema, even if this generated an XML verification fault when it arrived.

In fact, in many cases (where an XML framework like WCF supports this approach) the new element will never be detected in the first place.  But where it has been, it can be safely ignored, because since it was unexpected, it can’t affect the way in which the component will process the rest of the message.

2. Accept and Log unexpected values in an Enumerated List

This type of data inconsistency can occur even when the new version of the SIF specification merely extends the set of possible values in a previously existing list (ex: a new Discipline code). It is more problematic than the previous example because here the unexpected value cannot simply be ignored, since it makes it impossible for the legacy application to process an element on which it was relying.

This situation can be viewed as either a bug in the new application or a failure of the migration strategy for the new release, because ideally no legacy application should suddenly start receiving code values for any SIF-defined element it supports, which were undefined in the standard at the time the application was written.

In actual fact it may be neither.

Normative dependencies on external code sets from other standards such as ISO Country Names mean there are times where SIF needs to modify or add to such sets even when issuing a minor release.  Another problem is that the code sets SIF contains do not always fully encompass the set of values required by the site, and additional code values need to be introduced at installation time.

Even SIF-defined enumerated code lists evolve and have to be updated in the XML schema, and there are times where an equivalent backwards mapping for each new element into the previous value set is impossible.  Backward compatibility becomes even more difficult when Events are involved, because the same identical message is placed in the queue of all subscribers, some of whom may have been written to conform to an earlier version of the specification, and the rest to the current one.

So a legacy application may receive an enumerated code value it does not understand, because it was not defined in the version of the standard the application conforms to.  In this case, the recommended action for the Agent is to:

1.      1. Accept the message

       2. Insert a “Code Not Understood” (or equivalent) value in the XML element which is passed back to the application. 

3.       3. Post a LogEntry Event containing the sender, the element and the unintelligible value.

Acceptance of the message is critical.  In most cases (incoming Response or Event) there is no way to report a problem back to the sender anyway.  Rejecting the message at the ZIS level is not correct, since that implies there was something wrong with the message header, which is not the case here.

Since it can’t be rejected, the rest of the message should not be thrown away.  However by inserting a previously defined and “recognized” error value for the element, the application can still process the remainder of the data.

The problem should also be brought to the attention of the Zone Administrator.  That is achieved by posting the LogEntry Event, ideally only once per new and unknown value (to avoid the danger of flooding the system with identical error messages).

With the adoption of these two programming practices it is possible to write a SIF Agent / Application which is both backwards AND forwards compatible across SIF minor releases.