Slow Responses & Errors hosting Laravel API on Heroku - php

Question

I am having an odd experience with Heroku hosting my Laravel app/API
Locally everything looks good but recently requests are slow or timing out.
I still get results (most of the time) but the logs show the below;
2021-04-07T14:07:08.267681+00:00 app[web.1]: [07-Apr-2021 14:07:08] WARNING: [pool www] child 145, script '/app/public/index.php' (request: "GET /index.php") executing too slow (3.062416 sec), logging
2021-04-07T14:07:08.268185+00:00 app[web.1]:
2021-04-07T14:07:08.268269+00:00 app[web.1]: [07-Apr-2021 14:07:08] [pool www] pid 145
2021-04-07T14:07:08.268346+00:00 app[web.1]: script_filename = /app/public/index.php
2021-04-07T14:07:08.268538+00:00 app[web.1]: [0x00007f4cb9618040] execute() /app/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:112
2021-04-07T14:07:08.268729+00:00 app[web.1]: [0x00007f4cb9617fc0] execute() /app/vendor/laravel/framework/src/Illuminate/Database/Connection.php:343
2021-04-07T14:07:08.269002+00:00 app[web.1]: [0x00007f4cb9617f20] Illuminate\Database\{closure}() /app/vendor/laravel/framework/src/Illuminate/Database/Connection.php:671
2021-04-07T14:07:08.269207+00:00 app[web.1]: [0x00007f4cb9617e60] runQueryCallback() /app/vendor/laravel/framework/src/Illuminate/Database/Connection.php:638
2021-04-07T14:07:08.269390+00:00 app[web.1]: [0x00007f4cb9617da0] run() /app/vendor/laravel/framework/src/Illuminate/Database/Connection.php:346
2021-04-07T14:07:08.269589+00:00 app[web.1]: [0x00007f4cb9617d10] select() /app/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:2313
2021-04-07T14:07:08.269788+00:00 app[web.1]: [0x00007f4cb9617ca0] runSelect() /app/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:2301
2021-04-07T14:07:08.270038+00:00 app[web.1]: [0x00007f4cb9617bd0] Illuminate\Database\Query\{closure}() /app/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:2796
2021-04-07T14:07:08.270249+00:00 app[web.1]: [0x00007f4cb9617b30] onceWithColumns() /app/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:2302
2021-04-07T14:07:08.270443+00:00 app[web.1]: [0x00007f4cb9617a50] get() /app/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php:588
2021-04-07T14:07:08.270650+00:00 app[web.1]: [0x00007f4cb9617920] getModels() /app/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php:572
2021-04-07T14:07:08.270837+00:00 app[web.1]: [0x00007f4cb9617820] get() /app/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php:477
2021-04-07T14:07:08.270978+00:00 app[web.1]: [0x00007f4cb96177a0] all() /app/vendor/tcg/voyager/routes/voyager.php:39
2021-04-07T14:07:08.271163+00:00 app[web.1]: [0x00007f4cb96176d0] {closure}() /app/vendor/laravel/framework/src/Illuminate/Routing/Router.php:423
2021-04-07T14:07:08.271362+00:00 app[web.1]: [0x00007f4cb9617650] loadRoutes() /app/vendor/laravel/framework/src/Illuminate/Routing/Router.php:382
2021-04-07T14:07:08.271552+00:00 app[web.1]: [0x00007f4cb96175b0] group() /app/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php:261
2021-04-07T14:07:08.271709+00:00 app[web.1]: [0x00007f4cb9617500] __callStatic() /app/vendor/tcg/voyager/routes/voyager.php:133
2021-04-07T14:07:08.271894+00:00 app[web.1]: [0x00007f4cb9617470] {closure}() /app/vendor/laravel/framework/src/Illuminate/Routing/Router.php:423
2021-04-07T14:07:08.272080+00:00 app[web.1]: [0x00007f4cb96173f0] loadRoutes() /app/vendor/laravel/framework/src/Illuminate/Routing/Router.php:382
2021-04-07T14:07:08.272276+00:00 app[web.1]: [0x00007f4cb9617350] group() /app/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php:261
2021-04-07T14:07:09.811958+00:00 app[web.1]: 99.99.99.99 - - [07/Apr/2021:14:07:05 +0000] "GET /api/v1/mentors HTTP/1.1" 200 7278 "-" "PostmanRuntime/7.26.10
2021-04-07T14:07:09.819006+00:00 heroku[router]: at=info method=GET path="/api/v1/mentors" host=xxxxxxxxxxxxxx.herokuapp.com request_id=17d1f729-e3a2-41dc-8c92-f0b917623b74 fwd="999.999.999.99" dyno=web.1 connect=1ms service=4615ms status=200 bytes=7586 protocol=https
But the other times things are snappy and I just get these types of logs on a 1-2 response time;
2021-04-07T14:15:37.182510+00:00 app[web.1]: 99.99.99.99 - - [07/Apr/2021:14:15:36 +0000] "GET /api/v1/mentors HTTP/1.1" 200 7278 "-" "PostmanRuntime/7.26.10
2021-04-07T14:15:37.183580+00:00 heroku[router]: at=info method=GET path="/api/v1/mentors" host=xxxxxxxxxxxxxxxxxxx.herokuapp.com request_id=0b8ccc3f-8e3f-44bf-91c0-71ef6d245ad2 fwd="999.999.999.999" dyno=web.1 connect=0ms service=300ms status=200 bytes=7586 protocol=https
2021-04-07T14:15:47.637135+00:00 app[web.1]: 99.99.99.99 - - [07/Apr/2021:14:15:47 +0000] "GET /api/v1/mentors HTTP/1.1" 200 7278 "-" "PostmanRuntime/7.26.10
2021-04-07T14:15:47.638294+00:00 heroku[router]: at=info method=GET path="/api/v1/mentors" host=xxxxxxxxxxxxxxxxxxx.herokuapp.com request_id=8b415575-eb68-4fd4-87df-bca90f5fbf26 fwd="999.999.999.999" dyno=web.1 connect=0ms service=454ms status=200 bytes=7586 protocol=https
2021-04-07T14:15:51.658485+00:00 heroku[router]: at=info method=GET path="/api/v1/mentors" host=xxxxxxxxxxxxxxxxxxx.herokuapp.com request_id=c91c02d4-5141-4002-81aa-05507c0ea3f9 fwd="999.999.999.999" dyno=web.1 connect=1ms service=729ms status=200 bytes=7586 protocol=https
2021-04-07T14:15:51.655872+00:00 app[web.1]: 99.99.99.99 - - [07/Apr/2021:14:15:50 +0000] "GET /api/v1/mentors HTTP/1.1" 200 7278 "-" "PostmanRuntime/7.26.10
I thought it might be some queries with a number of relationships defined but even basic lookups on a single table is giving the same varied responses.
I also looked at the configureRateLimiting value and increased that from 60 to 10000 but nothing seems to change.
Anybody experienced similar? any ideas where to start to troubleshoot this weirdness?
Chris

Answer 1

I'm also having the same issue. However, this issue only occurs when requesting to the website the first time after I deploy changes to heroku. Subsequent requests after that are okay.
When slow execution is detected, function calls or scripts are logged. In your case the app starts to get slow when it reaches database connection part. Quoting from your log:
execute() /app/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:112
execute() /app/vendor/laravel/framework/src/Illuminate/Database/Connection.php:343
...
Perhaps you are trying to connect to a remote database? (other database provider aside from heroku). You can try installing composer package barryvdh/laravel-debugbar in your development environment to see how many queries you call, if there are any duplicates and how long does the queries took before loading the page.
Here is the logs on my end:
2021-11-18T06:46:17.662199+00:00 app[api]: Release v523 created by user bk2o1.syndicates#gmail.com
2021-11-18T06:46:18.304816+00:00 heroku[web.1]: State changed from down to starting
2021-11-18T06:46:18.355707+00:00 heroku[worker.1]: State changed from down to starting
2021-11-18T06:46:25.941551+00:00 heroku[worker.1]: Starting process with command `php artisan queue:work`
2021-11-18T06:46:26.562345+00:00 heroku[worker.1]: State changed from starting to up
2021-11-18T06:46:28.137180+00:00 heroku[web.1]: Starting process with command `vendor/bin/heroku-php-apache2 public/`
2021-11-18T06:46:30.999737+00:00 app[web.1]: DOCUMENT_ROOT changed to 'public/'
2021-11-18T06:46:31.213248+00:00 app[web.1]: Detected 536870912 Bytes of RAM
2021-11-18T06:46:31.279814+00:00 app[web.1]: PHP memory_limit is 128M Bytes
2021-11-18T06:46:31.279869+00:00 app[web.1]: Starting php-fpm with 4 workers...
2021-11-18T06:46:31.354604+00:00 app[web.1]: Starting httpd...
2021-11-18T06:46:31.868967+00:00 heroku[web.1]: State changed from starting to up
2021-11-18T06:46:59.375247+00:00 app[web.1]: [18-Nov-2021 06:46:59] WARNING: [pool www] child 146, script '/app/public/index.php' (request: "GET /index.php") executing too slow (3.248194 sec), logging
2021-11-18T06:46:59.375717+00:00 app[web.1]:
2021-11-18T06:46:59.375797+00:00 app[web.1]: [18-Nov-2021 06:46:59] [pool www] pid 146
2021-11-18T06:46:59.375873+00:00 app[web.1]: script_filename = /app/public/index.php
2021-11-18T06:46:59.376056+00:00 app[web.1]: [0x00007f6d6dc1b4a0] curl_multi_select() /app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:160
2021-11-18T06:46:59.376216+00:00 app[web.1]: [0x00007f6d6dc1b3f0] tick() /app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183
2021-11-18T06:46:59.376355+00:00 app[web.1]: [0x00007f6d6dc1b370] execute() /app/vendor/guzzlehttp/promises/src/Promise.php:248
2021-11-18T06:46:59.376501+00:00 app[web.1]: [0x00007f6d6dc1b2d0] invokeWaitFn() /app/vendor/guzzlehttp/promises/src/Promise.php:224
2021-11-18T06:46:59.376648+00:00 app[web.1]: [0x00007f6d6dc1b250] waitIfPending() /app/vendor/guzzlehttp/promises/src/Promise.php:269
2021-11-18T06:46:59.376797+00:00 app[web.1]: [0x00007f6d6dc1b1c0] invokeWaitList() /app/vendor/guzzlehttp/promises/src/Promise.php:226
2021-11-18T06:46:59.376940+00:00 app[web.1]: [0x00007f6d6dc1b140] waitIfPending() /app/vendor/guzzlehttp/promises/src/Promise.php:62
2021-11-18T06:46:59.377110+00:00 app[web.1]: [0x00007f6d6dc1b0b0] wait() /app/vendor/microsoft/azure-storage-blob/src/Blob/BlobRestProxy.php:3018
2021-11-18T06:46:59.377313+00:00 app[web.1]: [0x00007f6d6dc1b020] getBlobProperties() /app/vendor/league/flysystem-azure-blob-storage/src/AzureBlobStorageAdapter.php:240
2021-11-18T06:46:59.377510+00:00 app[web.1]: [0x00007f6d6dc1aef0] getMetadata() /app/vendor/league/flysystem-azure-blob-storage/src/AzureBlobStorageAdapter.php:153
2021-11-18T06:46:59.377670+00:00 app[web.1]: [0x00007f6d6dc1ae80] has() /app/vendor/league/flysystem-cached-adapter/src/CachedAdapter.php:212
2021-11-18T06:46:59.377801+00:00 app[web.1]: [0x00007f6d6dc1add0] has() /app/vendor/league/flysystem/src/Filesystem.php:58
2021-11-18T06:46:59.377981+00:00 app[web.1]: [0x00007f6d6dc1ad50] has() /app/vendor/laravel/framework/src/Illuminate/Filesystem/FilesystemAdapter.php:110
2021-11-18T06:46:59.378080+00:00 app[web.1]: [0x00007f6d6dc1ace0] exists() /app/app/Models/Image.php:39
2021-11-18T06:46:59.378317+00:00 app[web.1]: [0x00007f6d6dc1ac20] getOriginalUrlAttribute() /app/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php:527
2021-11-18T06:46:59.378532+00:00 app[web.1]: [0x00007f6d6dc1ab90] mutateAttribute() /app/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php:541
2021-11-18T06:46:59.378762+00:00 app[web.1]: [0x00007f6d6dc1ab00] mutateAttributeForArray() /app/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Concerns/HasAttributes.php:163
2021-11-18T06:46:59.378960+00:00 app[web.1]: [0x00007f6d6dc1aa60] attributesToArray() /app/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php:1403
2021-11-18T06:46:59.379155+00:00 app[web.1]: [0x00007f6d6dc1a990] toArray() /app/vendor/laravel/framework/src/Illuminate/Collections/Traits/EnumeratesValues.php:828
2021-11-18T06:46:59.379373+00:00 app[web.1]: [0x00007f6d6dc1a910] Illuminate\Support\Traits\{closure}() /app/vendor/laravel/framework/src/Illuminate/Collections/Collection.php:642
2021-11-18T06:47:26.126488+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/" host=my-website-url-obviously.herokuapp.com request_id=825cada4-f0e1-4019-9edf-963bb47c995c fwd="112.200.130.157" dyno=web.1 connect=0ms service=30000ms status=503 bytes=0 protocol=https
2021-11-18T06:47:26.420440+00:00 app[web.1]: [18-Nov-2021 06:47:26] WARNING: [pool www] child 146, script '/app/public/index.php' (request: "GET /index.php") execution timed out (30.290064 sec), terminating
2021-11-18T06:47:26.983257+00:00 app[web.1]: 10.1.16.72 - - [18/Nov/2021:06:46:56 +0000] "GET / HTTP/1.1" 200 20063 "https://dashboard.heroku.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36 Edg/95.0.1020.44
Which consists of calls using guzzle since I was fetching images from Azure:
getBlobProperties() /app/vendor/league/flysystem-azure-blob-storage/src/AzureBlobStorageAdapter.php:240
exists() /app/app/Models/Image.php:39
And on my App\Models\Image line 39, I was checking if image exists before actually fetching it. The issue came up when I eager loaded 62 images:
// in App\Models\Image.php
if(Storage::disk($this->disk)->exists($path.'/'.$filename)) {
// fetch the thing here
}
Solved by avoiding the check and just returning a generated url using storage facade
// in App\Models\Image.php
return Storage::disk($this->disk)->url($path.'/'.$filename).'?'.$this->updated_at?->format('U');
Another thing to consider is how much traffic your app is receiving when things started going slow. Data being displayed on page dynamically can really take up huge amount of memory depending on how much data is kept on memory during runtime and how many visitors are requesting for data (note: EVEN IF THE DATA IS CACHED). From my experience I also noticed sudden slow response times when bots are visiting the app.
This is not really a way to solve your issue but more of an overview of what might be going on under the hood. And most importantly the logs gives us a hint to what is happening and you can go from there.