ProPublica

Problem

There is comprehensive doctor discipline data available across 50 different state medical board sites. However, it is not easy to look at standardized data across the different state websites. The public has no way to know if their in-good-standing physician faced disciplinary action in a different state. Prior reporting has shown that many doctors move from state to state after receiving a disciplinary action as a way to cover their tracks.

Solution

ProPublica wanted to create an app with standardized, comprehensive data about their doctors across the states.

We used python package selenium for scraping and did a few states. We also incorporated multithreading which enabled multiple states to be scraped simultaneously. States had disciplinary files (sometimes hundreds which is why the multithreading was important) associated with doctors as well, so also incorporated aws to store these

See our code

About the Project

Technical Challenges

Biggest challenge was that this was very different from prior projects we’d taken on, given that it didn’t really follow the general flask base boilerplate we had at the time. Selenium and multithreading were new to most team members and figuring out how to efficiently get all the data was also a challenge.

About the Client

ProPublica’s mission is as follows: “To expose abuses of power and betrayals of the public trust by government, business, and other institutions, using the moral force of investigative journalism to spur reform through the sustained spotlighting of wrongdoing.”