EveryPolitician bot

Our little bot works tirelessly collating the EveryPolitician data and keeping it up to date.

OK, so in fact it’s not really a robot, and it doesn’t look like this. Nonetheless, if you’re interested in the some of the implementation and design details behind the project, we think the bot’s point-of-view might be a good way to explain things.

• the bot’s blog is on Medium
• the bot tweets as @everypolitician

Blog posts

• Introducing the bot

  An introduction to the bot (on mySociety’s blog).

• EveryPolitician as a pipeline

  The bot’s high-level overview of our data-building process: from multiple sources to useful CSV and JSON files.

• I am a busy bot

  The bot makes lots of commits and pull requests, triggered by webhooks.

• I’m a bot who comments

  We use the bot to inspect pull requests and add summaries to them as comments.

• I make lists of humans’ names

  About names.csv, and an example of an app being kept up-to-date by using EveryPolitician webhooks.

• Getting busy with scraper data

  How the data gets from multiple scrapers into the EveryPolitician data repo.

• I am a (pull request) terminator

  When the bot generates a new data pull request, it automatically tidies up previous ones.

• I keep the index up to date

  How we use GitHub webhooks to add SHAs to countries.json, keeping it timely.

• I ❤ webhooks: pass it on

  We’ll make a POST request to your URL whenever the EveryPolitician data changes.

• How I build the EveryPolitician website

  The bot makes EveryPolitician’s (static) website by spinning up a dynamic one and spidering its pages into gh-pages.

• I get versioned deploy logs for free

  A simple benefit from using GitHub Pages to host the EveryPolitician website.

• I let humans peek into the future

  The bot detects each new data pull request, and spins up a preview site on Heroku showing how it will look.

• I let humans peer into the past

  The same mechanism for previewing and building the live site lets us look at old commits too.

• I’m a well-behaved friend of the Octokit

  Using GitHub’s library wrappers to access the GitHub API.

• I’m the good kind of terminator

  We reprogrammed to bot to stop it closing pull requests that humans are working on.

• I have busy days

  The bot really can make a lot of commits and pull requests.

• Sometimes I work hard to produce nothing

  Using git as a data store, and how it’s OK that we rebuild the data just to determine nothing’s changed.

• How I avoid the identity crisis

  EveryPolitician adds its own UUIDs, but also retains useful IDs from external sources.

• I let humans have the final word

  Although the data pull requests are usually prepared by the bot, a human decides whether or not to merge them.

• I use Wikidata for multilingual names

  Wikidata is a wonderful, always-updating source of international transliterations.

• I merge multiple sources

  The bot merges multiple sources into a single set of data for every legislature.

• I work the full multi-bot 24-hour shift

  A look at the event-driven nature of the EveryPolitician system.

• I import data in CSV format

  We needed to import CSVs because people use spreadsheets, and we use the same simple format for more complex imports too.

• My instructions are metadata. In JSON.

  Everything the bot needs to know in order to combine data from different sources is in the file instructions.json.

• Introducing the EveryPolitician gem

  The easy way to access and manipulate the data in Ruby: use the gem.

• My data can boost your data: Politwoops example

  An example of how EveryPolitician data can simply augment your data, for free.