Birkin James Diana, Brown University

In 2007 we built a system that dumped certain user web-requests for books into a database for offline-processing triggered via cron. We wanted to make the magic happen live, but knew it would take too long. Thus we created, sort of accidentally, a kind of old-fashioned static procedural job queue.

Over the years we we’ve been repeatedly impressed with how useful and robust this unintended architecture has been, and it fostered thinking about using real job queues in Library workflows.

Fast-forward to the present. We now are using real job queueing, in production, for parts of the functioning of Brown Digital Repository. We’ve also used it for ingestion scripts, and plan to move more lots more code to this architecture.

I’d like to share & show:

  • our lightweight rq/redis job queueing setup
  • how using job queues can speed up workflows via using multiple workers
  • how job queueing can make workflows more robust, especially by simplifying failure handling
  • a way we've smoothly avoided race-conditions that can occur in concurrent-programming
  • a technique for using task-processing job queues to simplify complex workflows

rq: http://python-rq.org redis (python): https://pypi.python.org/pypi/redis/