This article is part of the CakeDC Advent Calendar 2024 (December 17th 2024)
This is a small tip related with an issue we fixed not so long ago. While executing a report, exporting the data as a csv file, we hit a max_memory issue. That's a very common issue when your tables grow bigger and the amount of data you need to export does not fit into the available memory for the PHP process.
In that case you have a couple options like:
- Push the report generation to a background job, then send the result link via email
- Work with ob_start(), flush() etc
But today we are going to show a way to do it using CakePHP builtin methods.
Let's imagine we are exporting a big Users table like:
$users = $this->fetchTable('Users')->find();
foreach ($users as $user) {
// handle the user row
}
Causing a 500 error due to a lack of memory to store all the users resultset.
And we don't want to process anything in the background, we just want the user to "wait" for the result being generated (note this also could be an issue if it's too long).
We could use the following approach in our controller:
return $this->response
->withType('text/csv') // or other formats, in case of xml, json, etc
->withDownload('my-big-report.csv')
->withBody(new \Cake\Http\CallbackStream(function () {
$users = $this->fetchTable('Users')->find()
->disableBufferedResults()
->disableHydration()
->toArray();
$outstream = fopen("php://output", 'w');
foreach ($users as $user) {
fputcsv($outstream, $row);
}
}));
A note about the solution
disableBufferedResults
does not work for all datasources due to a limitation in PDO. It works for MySQL.- In case your are using a not compatible datasource, you'll need to paginate the query manually to get the results in chunks
So, using this approach we are paying time (more queries to the database, longer response time, user waiting) for RAM. Depending on your case, it could be a fair tradeoff.
Marc (@ravage84) suggested in slack to also take a look at https://github.com/robotusers/cakephp-chunk too for a way to chunk the results of a query, processing them in a memory efficient way.
This article is part of the CakeDC Advent Calendar 2024 (December 17th 2024)