ActionMailer is great. It allows you to create email templates and put logic in Mailer classes. You can use Roadie to merge in CSS, customizing look and feel. The problem arises when you have to send tens or hundreds of thousands of emails. Each one is a separate API or SMTP call to your email service provider.
We were able to achieve some perf gains by breaking up our jobs in smaller batches and then running multiple processes per server with ActiveJob. But the HTTP REST or SMTP calls are still very time consuming when you are doing one per email.
Alternatively if you build email templates in the Mandrill or Sendgrid UI you can then call their API passing smaller hashes with the appropriate parameters (first_name, last_name, etc) and they will do the appropriate substitution. That approach is much faster but you loose a lot of flexibility of being able to build your emails in code. Plus someone could change email template in Mandrill UI and you have no revision control.
So here is how we recently attempted to solve it (it’s still work in progress).
Create appropriate templates for this mailer in app/views.
Created SendEmailsJob in app/jobs. It can be run as ActiveJob via Sidekiq, DelayedJob or another queue.
Create EmailInterceptor class. You can put email_interceptor.rb in mailers folder.
You will need to put this in an initializer
Create a template in Mandrill or Sendgrid with one basic variable like
Create SendEmailBatchJob which actually sends emails via API call to the email service provider. The problem is that queueing up these emails will take TONS of RAM in Redis because you are storing the actual HTML (could be tens of kilobytes per message once you include the fancy CSS). So unless you have a huge Redis instance just sitting around, it’s not a good approach.
You can use AWS SQS and Shoryuken but what if you do not want all your jobs going through SQS? Fortunately I found this issue https://github.com/rails/rails/issues/16960 which allows you to configure queueing systems on per Job class.
- Set config.active_job.queue_adapter = :sidekiq in application.rb (or production.rb). This way your regular jobs will run through Sidekiq and only run these huge batches of emails through SQS.
- Login to AWS SQS web console and create the each_email queue. Keep in mind the AWS SQS max message size is 256KB but that should be enought.
You can spin up multiple workers/threads to run SendEmailBatchJob across different servers or even setup separate cluster. All they do is grab jobs from SQS and call email service provider API.
We are still in the process of rolling it out and testing so I will post updates on our progress.
For those who are not running Rails 4.2 there is ActiveJob Backport which works great.