Previously I blogged about creating Microservices with Sidekiq. This artcile is an expansion on those ideas.

Most of us are familiar with how Google Analytics or Mixpanel track user interactions on websites. Or perhaps we used error notifcation services such as Rollbar, RayGun or AirBrake. The common pattern is to separate the system into one component that simply receives the messages and another component(s) that process the data and display it to the users.

To demo these concepts I built a sample app. Requests contain user’s IP, User Agent, Time of the event and URL (which sent the request). It is oversimplified to show the basic functionality.

API

It has a simple endpoint in HomeController that takes requests params and throws them into Redis queue using Sidekiq. It is built using Rails 5 API but could be implemented with Ruby Sinatra, NodeJS / Express or Python Flask. There is a Sidekiq implementation in nodejs or you could build your own client to throw messages into Redis in the appropriate format.

class HomeController < ApplicationController
  def index
    job_params = request.params.except(:controller, :action)
    ProcessRequestJob.perform_later(job_params)
    render status: 200
  end
end
class ProcessRequestJob < ApplicationJob
  queue_as :default
  def perform()
    # simply queue the job
  end
end

API is completely unaware of the main DB or any other components. API ProcessRequestJob does not actually do anything. It is simply an easy way to queue the job with .perform_later call from the HomeController. Sidekiq background process does not run w/in API.

After cloning the repo you need to cd api && bundle && rails s. API also contains api.rake tasks which makes HTTP requests to http://localhost:3000 passing various IPs, user agents, etc. Just run rake api:test_requests.

UI

It is build with Rails 5 / ActiveRecord / SQLite. It uses RailsAdmin CRUD dashboard so you can view the Requests table and see how data is aggregated. There is basic authentication / authorization with Clearance, Pundit and Rolify.

UI also contains the Sidekiq library that actually processes background jobs. In true microservices design it would probably be a separate application. ProcessRequestJob class contains the biz logic for grabbing parameters stored w/in each job and creating records in the main DB in Requests table.

class ProcessRequestJob < ApplicationJob
  queue_as :default
  def perform(*args)
    # biz logic for processing records, could be moved to separate class
    params = args.first
    Request.create(ip: params[:ip], time: params[:time], url: params[:url], user_agent: params[:user_agent])
  end
end

After cloning the repo you need to cd ui && bundle && rake db:seed && rails s -p 3001

Browse to http://localhost:3001/ and login with admin@email.com / password. You can then access http://localhost:3001/admin CRUD dashboard and http://localhost:3001/sidekiq. You will see how background jobs queue up when you run rake task from api folder. Then run sidekiq in the ui folder and it will process jobs creating Request records.

Design

This approach would allow us to scale API or UI separately, upgrading or replacing components as needed. We could rewrite API endpoint in different framework, deploy it to production and do realistic A/B performance test.

We could upgrade Redis servers or even replace Redis with a different queue such as AWS SQS. We would deploy the upgraded API and it will start pushing messages to the new queue. Then we wait for old Redis queue to drain (should not take long) and deploy the new background job processor code. As long as the message format remains the same the background job process and API will function independently.

We could even replace our main DB (move from MySQL to Postgres). That would require downtime for the UI while data is migrated but the API will be simply collecting data in the queue. After migration you will need to update the DB connection string in the the background job process and start it up. Just be careful NOT to exceed RAM needed for Redis.

None of the ideas I described above are revolutionary. What I like about this approach is how easy it is to integrate separate applications and quickly build a very robust and flexible system.