Batch processing with Symfony and Doctrine

Chances are that you have encountered with “memory” error when working with data through Doctrine ORM. In fact, an ORM (Object-relational mapping) is not the best tool to use when processing a lot of data, however it’s a lot easier for developers to process entities massively, because, well, you most likely have already created the entities and you just need to use ORM functions to process them the way you want.

The problem in bulk operations using ORM is that the objects which are hydrated (each row from database need to be mapped to entity object), are stored in memory, and the same happening when we are creating new entities and storing them to database through entity manager. Because of that, we getting to the peak of memory usage and the error message is something like: “Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 72 bytes) in …“.

So to keep things short, what would help us – is to detach objects from Doctrine, or in other words – clearing them from memory.

Bulk Processing

They key in optimizing memory when processing mass data is to use iterable result and detaching when we are done doing some stuff with the current iterable object. In other words, it’s called step-by-step hydration.

 

Bulk Inserts

When inserting a lot of objects, a good approach would be to set batch count, which we could use to track when to flush and clear objects from Doctrine, like so:

You can read more in this great Doctrine article about batch processing in Doctrine.

Leave a Reply

Your email address will not be published.

*
*