Effectively evaluating the appropriateness of clinical decision support (CDS) alerts and responses is critical to improving patient safety through health information technology. We describe the use of REDCap to create a data collection instrument for evaluating the appropriateness of CDS alerts and responses. While the tool is effective, further enhancements to REDCap would improve the process for future evaluations.