JAMIA: Creative solutions needed to overcome NLP barriers
Although natural language processing (NLP) research in the clinical setting has occurred since the 1960s, progress in developing NLP applications for clinical text has been slow and lags behind progress made in the general NLP domain, according to an editorial in the September issue of Journal of the American Medical Informatics Association.
Wendy W. Chapman, PhD, from the department of biomedical informatics at the University of California, San Diego, and colleagues wrote there are several barriers to clinical NLP development, and shared tasks like the i2b2/VA Challenge address some of these barriers. However, barriers remain “and unless the healthcare community takes a more active role in developing new approaches for addressing the barriers, advancement and innovation will continue to be slow,” the editorial stated.
Historically, there have been barriers to NLP development in the clinical domain, though these barriers are not unique, as they also occur in software engineering and general NLP.
In the clinical context, Chapman et al stated that hospitals and clinics have been reluctant to allow access to clinical data from outside researchers, because of patient privacy concerns and possibly revealing unfavorable institutional practices. There has also been a reluctance to share data because there is no reliable and inexpensive de-identification technique for narrative reports.
“Such restricted access to shared datasets has hindered collaboration and inhibited the ability to assess and adapt NLP technologies across institutions and among research groups. Several pioneering efforts have made clinical data available for sharing—we need more of these grass-roots efforts,” the editorial stated.
Without the ability to share data, the authors wrote, the community lacks incentive to develop common data models for manual and automatic annotations.
The result: annotated datasets are small and unique to the lab that generated them; and NLP modules that perform the same tasks cannot be substituted and compared without considerable translation effort. However, the authors said the clinical NLP community is now using existing standards and conventions, to develop shared data models that map annotations across information extraction applications.
One partial solution being used is shared tasks such as the i2b2/VA Challenge, which addresses several of the barriers. Shared tasks provide annotated datasets to participants and sometimes non-participants (i2b2 datasets are available to others one year after the challenge). The i2b2 shared task is standardized—the same records are used from one year to the next with annotation layers built on each other, and common input/output specifications applied each year.
Shared tasks provide value to NLP in several ways:
Although shared tasks contribute to the growth, progress and increased benefit to the clinical NLP developer community, potential users need additional support to creatively target the existing barriers, the authors stated.
The shared i2b2 evaluations have contributed to stimulating and vitalizing clinical NLP; however, to ensure a transition into usable applications, the clinical NLP research community needs to address the issues of data access, development of a shared infrastructure and integration of software engineering methods for usability and availability.
“This must be done in collaboration with end users, software engineers, and clinicians,” Chapman and colleagues concluded. “We, as a community, need to think beyond the status quo of incremental improvement toward imaginative approaches that encourage collaboration, promote reproducibility, increase the scalability of NLP development, and provide value to end users.”
Wendy W. Chapman, PhD, from the department of biomedical informatics at the University of California, San Diego, and colleagues wrote there are several barriers to clinical NLP development, and shared tasks like the i2b2/VA Challenge address some of these barriers. However, barriers remain “and unless the healthcare community takes a more active role in developing new approaches for addressing the barriers, advancement and innovation will continue to be slow,” the editorial stated.
Historically, there have been barriers to NLP development in the clinical domain, though these barriers are not unique, as they also occur in software engineering and general NLP.
In the clinical context, Chapman et al stated that hospitals and clinics have been reluctant to allow access to clinical data from outside researchers, because of patient privacy concerns and possibly revealing unfavorable institutional practices. There has also been a reluctance to share data because there is no reliable and inexpensive de-identification technique for narrative reports.
“Such restricted access to shared datasets has hindered collaboration and inhibited the ability to assess and adapt NLP technologies across institutions and among research groups. Several pioneering efforts have made clinical data available for sharing—we need more of these grass-roots efforts,” the editorial stated.
Without the ability to share data, the authors wrote, the community lacks incentive to develop common data models for manual and automatic annotations.
The result: annotated datasets are small and unique to the lab that generated them; and NLP modules that perform the same tasks cannot be substituted and compared without considerable translation effort. However, the authors said the clinical NLP community is now using existing standards and conventions, to develop shared data models that map annotations across information extraction applications.
One partial solution being used is shared tasks such as the i2b2/VA Challenge, which addresses several of the barriers. Shared tasks provide annotated datasets to participants and sometimes non-participants (i2b2 datasets are available to others one year after the challenge). The i2b2 shared task is standardized—the same records are used from one year to the next with annotation layers built on each other, and common input/output specifications applied each year.
Shared tasks provide value to NLP in several ways:
- Common evaluation metrics are developed.
- Annotated datasets are made available.
- Researchers in overlapping fields (both academic and corporate) participate in the tasks, bringing in new people and approaches.
- Benchmarking on a shared dataset reveals the state-of-the-art performance for a given task.
- Preliminary results can be obtained by a new research group, which can lead to possible funding opportunities.
- Pre-processed, standardized tasks with multiple layers of annotations pave the way for evaluations.
- Conventions for standardizing annotations and input/output formats are developed, and despite other standardization, shared tasks often set the de facto standards.
Although shared tasks contribute to the growth, progress and increased benefit to the clinical NLP developer community, potential users need additional support to creatively target the existing barriers, the authors stated.
The shared i2b2 evaluations have contributed to stimulating and vitalizing clinical NLP; however, to ensure a transition into usable applications, the clinical NLP research community needs to address the issues of data access, development of a shared infrastructure and integration of software engineering methods for usability and availability.
“This must be done in collaboration with end users, software engineers, and clinicians,” Chapman and colleagues concluded. “We, as a community, need to think beyond the status quo of incremental improvement toward imaginative approaches that encourage collaboration, promote reproducibility, increase the scalability of NLP development, and provide value to end users.”