rails_admin_import is a great gem allowing us to import records. But sometimes we have to import many thousands of records and this gem does not scale well. What we want to do is display “import has began” message to the user and queue up a background process to import records. Here is a pattern I have been following.
Let’s imagine a multitenant system where Users belong to multiple Clients via UserClient relationship. We want to be able to import Users, Clients and UserClients.
Create bgimport custom action:
We need a basic UI to upload files with data.
We will be running jobs in a batch and using Redis to store job IDs:
Create PORO service object. This could be moved into Bgimport custom action class but it’s easier to test in a PORO. It will queue up individual jobs for each row in spreadsheet. It will use naming conventon pattern to determine which job class to call.
Create appropriately named jobs:
This approach gives us a lot of control on how to implement biz logic specific to our application. For example, let’s say that users must have unique emails within clients. We can query DB by those 2 params and then create or update user record. In case of UserClient we need to first find User and Client records (perhaps User by email and Client by name) and then create/update UserClient record.
When importing large amounts of data we are likely to encounter different errors with some of the records. We want to give users valid feedback. I describe various options in this post
Separately we might want to import data not via UI file upload but by downloading records from FTP / API. We simply create another job to download the file and pass it to the QueueImport.