I am a Sr. Software Developer at Oracle Cloud. The opinions expressed here are my own and not necessarily those of my employer.
Redis as temp cache for application-side joins
Oct 11, 2016
SQL joins are a powerful feature that enables using DB functionality to bring back records from different tables needed w/o making multiple queries. Unfortunately some of the new NoSQL DBs do not support them.
We have been using MongoDB for several years and overall really like it. Except that sometimes we have to do N+1 queries looping through child records and fetching data attributes from the parent or grandparent.
Let’s walk through implementing application side joins where we are combining data attributes from different queries. Our example is a CMS application built with Ruby on Rails and Mongoid.
Now we need to generate a report of recent Comments and include Article titles and names of User who wrote the articles. In SQL we could write:
With SQL DBs we can eager load associations which will use joins to get Users. But with Mongo we can only do Comment.recent.includes(:article) to eager load Articles and we end up querying for EACH user.name separately.
What if we could fetch all Articles for recent comments and then fetch all related User records? It will be more than 1 query but it is better than N+1.
We do not need all User and Article attributes so we specify fields using .only. But where do we store the User and Article records as we loop through Comments while generating our report? We could build our own data structures but why not throw them into Redis as Hashes?
Data cached in Redis will look like this:
57fc62651d41c873ba6c880c is part of User key AND is stored as user_id in Article hash. Now we can loop through Comments but instead of using regular relationships c.article.body and c.article.user.name (which would have caused DB queries) we are grabbing data attributes from records cached in Redis. If we use Comments.includes(:article) then we only need Redis for User records caching.
Last we remove the records cached in Redis.
This design requires writing more code but it speeds up your report generator AND decreases DB load. It can also work when querying records from multiple SQL DBs or from 3rd party APIs. We fetch each dataset separately and use Redis as temp data store.