The Case for Continuous Interoperability

Interoperability has been a key buzz word heard along the road toward modernizing the American healthcare system. But standards, conformance to those standards, and thorough testing to ensure conformance are all required to achieve true, seamless information exchange. That’s the message advocated by Mario Hyland, senior vice president and founder of AEGIS, a consulting firm focused on advancing a health IT testing infrastructure. Hyland spoke with Clinical Innovation + Technology about current interoperability and testing challenges.

“Interoperability is not just about data repositories and being able to move data,” he said. “There’s so much more. It continues to be hard and we need to do more testing. The only way to ensure interoperability is through testing.”

CIT: What do you think needs to happen regarding testing of health IT?

MH: We’re happy the Office of the National Coordinator for Health IT (ONC) published a 10-year plan for achieving interoperability. However, there are some health IT-focused organizations making claims such as “80 percent or more” of interoperability issues have been solved and we don’t agree. ONC’s 10-year plan is a step in the right direction, but it doesn’t address some of the technical challenges we see. The term ‘interoperability’ from our perspective is a state in time. There are no guarantees that if a system is interoperable today it will be tomorrow--yet it absolutely needs to be. Patient safety, data security, quality of care--all these things and more depend on it. So we need to change that mindset into one we deem ‘continuous interoperability.’ Our approach to supporting that goal has been to develop an automated test platform, identify and document best practices and, ultimately, support an environment for reuse of tools to go beyond “happy path” testing that’s all too often the most that gets done today.

CIT: Does testing need to be part of certification?

MH: As I mentioned, some people mistakenly feel interoperability is solved. As a result, testing is often viewed as just a subset of a certification process. In other words, vendors want to do just enough testing to get certified. That’s not the right mindset. We need to start by recognizing that some challenges around interoperability begin with the definition. When we talk interoperability, we’re talking system to system interaction—the ability of the receiving system to recognize and semantically understand the data being sent and, more importantly, the intent and purpose of that data from the perspective of the initiating or sending system. That’s the key to interoperability and it can only be achieved by implementing standards and ensuring that the sending and receiving entities both have implanted those standards correctly. Testing’s role in a certification process is to ensure that the standards have been interpreted and implemented correctly. So yes, testing has an important role in certification--a more important role than it’s being given today in some situations.

CIT: How can healthcare ensure that has happened?

MH: It hinges on standards. Healthcare (and any other discipline of critical, repeatable processes when you think about it) depends on standards. The question is which standards to implement? There are multiple standards out there right now. As ONC continues to issue guidance, we’re hoping they include a roadmap for developing EHR systems that ensure interoperability based on the right standards. That roadmap doesn’t exist.  When a CMIO needs to make an investment decision it would be great if ONC provided guidance along the lines of, “here are standards your EHR needs to have today and here are ones we expect it will need to support tomorrow.”

Getting the standards right is another challenge. After the implementation stage for a new standard and during the subsequent testing between organizations we begin to test the interoperability of a standard. What AEGIS has seen is that testing at that point often reveals variation in how different organizations interpret and implement the same standard. We’ve seen cases where 80 percent or more of organizations fail on a particular standard. We need to collect those test results and provide a feedback loop to the standards bodies (SDOs) giving them actionable data to evaluate and possibly refine the most problematic standards preventing interoperability. You would like for any organization to independently read a standard, implement, conduct testing and be found to be interoperable. That’s the ideal state, but the fact is that it’s not happening that way today. We know that testing can play a role in helping to get to that ideal state.

CIT: Which of the standards bodies do you feel are “doing it right” with respect to testing?

MH: There are a few that come to mind: Healtheway, a private-public partnership that assumed the governance role from the ONC NwHIN, now manages the eHealth Exchange, a national exchange with on-boarding processes and programs focused on interoperability among participant and vendor organizations.  Their on-boarding tests go beyond “happy path” and include “provisional testing” which introduces vendors to future tests which are coming online.  That’s very progressive and exactly what needs to be happening.

HL7’s new Conformance Testing Program was launched at the HIMSS national conference earlier this year. It was based on a Meaningful Use requirement, specifically the CDC Immunization Implementation Guide. HL7 recognizes the value in advancing testing in alignment with the development of standards.  IHE USA annually holds the Connectathon, which will be held in Cleveland in February 2015. The current Connectathon testing approach is limited to peer-to-peer testing which has its pros and cons, but generally speaking the Connectathon is a valuable testing venue.

CIT: Do you think vendors should take on the burden of testing?

MH: Vendors should definitely be testing, but buyers need to make sure the vendor is doing that testing. They need evidence--whether that’s a certification, independent test results or the buyer organization’s own testing. The buyer needs to know that patches, fixes and upgrades will be released, and how those will be tested to make sure interoperability is continuously maintained. Organizations with multiple systems or different modules from different vendors should be especially diligent to ensure that the vendors are testing and demonstrating interoperability, and they should consider their own independent testing as well to confirm that the testing is thorough and realistic.

Take the annual IHE Connectathon where software engineers test the heck out of health IT software. As I mentioned earlier, most of what happens there is “happy path” testing. In other words, testing peer-to-peer for what is supposed to happen. Peer-to-peer testing has its value and we need to do it but it’s not enough. To test an exception, something outside of the norm (i.e., things that are not supposed to happen, but in fact do happen in real life) would have to happen. In peer-to-peer testing, you would have to purposely break your system in a manner conducive to the test you want to try. Any one organization or development manager would not allow development resources to spend time purposely breaking their system so some other organization can test negatives or exceptions. That just doesn’t happen.

So, you may come out of an IHE Connectathon able to say that you’ve demonstrated happy path interoperability, but you are not in any position, based on a Connectathon alone, to claim that your system is production ready--yet that’s what industry feels.

So back to the question: should vendors take on the burden of testing? The problem is that they believe right now that the level of interoperability testing they do to obtain certification is enough and that they don’t need to do any more. Unless the buyers in the provider organizations implementing these solutions call for "more extensive testing" or make demands around it I’m afraid the vendors are not going to change. Some vendors are doing a good level of testing but not industry-wide testing. AEGIS has a platform—the Developers Integration Lab (DIL)--where this type of testing can be supported in a development environment and organizations don’t have to wait for implementation to test. We should and want to inject this type of testing into development.

Q: What can or should healthcare delivery organizations do to improve interoperability?

MH: The complexity of these systems and the need to have the ability to test are going to drive organizations, whether vendor or participant, into finding an environment that creates an integrated testing ecosystem. If you want to test your system in a transition of care (ToC) environment, you need an environment where lots of organizations (specifically, their systems) are available to test with. I believe that more testing needs to be done by vendors before they ship, as they make changes--changes made not only by that vendor but also when their partner organizations make changes. Again, continuous interoperability means you stay interoperable both when you make changes and when any of the systems you interoperate with make changes. As an industry we have to embrace this concept. The alternative is that we claim interoperability at a point in time, and when interoperability breaks down in production, we’re left with finger-pointing. How does that help the patient dying in an emergency room at 3am?

The message we’re trying to get out there is that an organization buying an EHR needs to know that vendors need to do more testing to demonstrate that they are interoperable today and that they actively monitor (test) to ensure interoperability continuously.

We are not in favor of certification alone, because it brings in a mindset of one and done. If a company gets certified they don’t want to have to get their product tested again next month. For interoperability, that represents an exposure. We want vendors and participants testing their systems all the time so we can statistically evaluate the maturity of the industry against the test case. When enough of industry demonstrates they understand the test and can pass it, we can take that test and nominate it as a representative test that could be part of certification. In other words, we need to “test the tests” before they become a requirement of certification. The bar will continue to rise as more and more tests are run.

Finally, it’s important to note that no certification programs are intended to be so comprehensive as to guaranty that a particular system is 100 percent production ready. We need other mechanisms for that. We want to help the vendors create something as simple as a report card that shows how often they test and when they find a problem how quickly do they fix it. That brings more value to the industry than a snapshot certification process with 50-100 tests that are a sampling of how an EMR operates and doesn’t convey that it’s production ready.

CIT: What do you think of ONC’s stance on testing?

MH: ONC says we should have a marketplace of testing tools. We believe the industry already has enough tools out there today called “box” tools that test small snippets of functionality. What AEGIS believes the industry really needs is more end-to-end testing.

Any development manager knows if you find an issue early and fix it at the development level it’s a lot cheaper than when you find problems close to or even after going into production. As we build standards we need to build the tests that exercise those standards at the same time. When standards are published, the tests need to be published as well allowing testing to begin as early as possible and repeated as often as possible. That’s AEGIS’ interoperability mantra: “Test early and test often.”

With quality healthcare depending on better ToC we need to support testing moving to an integrated environment that more closely represents what systems will encounter in the real world as patients go through these transitions. AEGIS is advancing the idea of an integrated ecosystem that supports testing at all levels with consistent rules-based conformance checking. We want to continue to raise the bar for vendors to demonstrate their quality advancements and ensure interoperability in a continuous testing environment. Global testing against a common platform provides SDOs critical feedback on their standards implementation and adoption rate while offering visibility into the maturity of their standards. Finally, we want to see certification programs based more and more on tests which have proven themselves as effective in properly and fairly exercising the standards. While we feel there is still a big hill to climb for achieving continuous interoperability among health IT systems, we see testing as the vehicle that will get the industry there. The good news is that the right tools are already in place today and ready to use.