This document discusses using crowdsourcing to teach machines medical diagnosis by having crowds annotate medical text. It describes several natural language processing tasks needed for medical question answering like entity identification and relation extraction. It also discusses challenges like disagreement between annotators and how the CrowdTruth framework represents disagreement to help machines learn. Experts are motivated to annotate through a game format. The goal is a system that harnesses both crowds and experts to teach machines.