As part of gradual refactoring we are undertaking here at carwow, we wanted to split a fairly large Rails app up into two smaller apps. Rather than do this as one big refactor, we chose to make things more manageable and less risky by doing it in small chunks. Unfortunately this meant that both apps would need to share the original PostgreSQL database for a period of time. Now generally this is probably not a good idea, but for us it was a temporary measure while we carried out the work.

One thing we knew we needed to be careful about was to ensure that any schema changes we made would work with both apps. This is just an extension of our normal approach of performing schema changes in multiple steps.

For example, when adding a new column, we would first add the column allowing it to be null. Then we would update the code to start populating the column and update existing records with suitable values. At this point we can update the new column to not allow null.

We run our migrations as part of our deploy and so in general migrations are run and the app is restarted around the same time. This approach has worked fine for us for a long time. Unfortunately we hit an unpleasant gotcha with our shared database approach.

What we saw when tried to add a new column recently was that our original app continued to work fine, but requests to the new app seemed to have hung and eventually were timing out. We then started getting the following errors.

ERROR:  cached plan must not change result type

After a little bit of digging it seems this is a known issue for Rails https://github.com/rails/rails/issues/12330. The short version of this is that we end up with cached prepared statements which are no longer valid since the underlying schema has changed. ActiveRecord tries to resolve this by deallocating the prepared statements, but if this happens inside a transaction, the deallocation fails since the transaction is already invalid from the previous attempt to use the statement . This results in the connection pool of ActiveRecord keeping a bad prepared statement and bad things follow.

For us the issue was caused by us not restarting the second Rails app after the migrations were run. We now avoid this issue by always restarting both apps after a migration, ensuring any cached statements are freed.

The good news is that this issue has been fixed on Rails master (https://github.com/rails/rails/pull/22170) but in general restarting your app after migrations is a good idea and if you ever have to have multiple apps sharing the same database don't forget to apply the same approach.

Interested in making an Impact? Join the carwow-team! Feeling social? Connect with us on Twitter and LinkedIn :-)