It’s been almost 4 years since my last post – yikes! I guess it’s time to dust off the cob webs and share something which took me many hours to figure how to run Supervisor with Laravel on Heroku.
Assumptions
Here are some assumptions that I’ve made that you know about before reading this article:
- Heroku – deployment, dynos and Procfile.
- Supervisor
- Laravel Queue Workers
The Problem
I’m running my application on Laravel 5.4 and using Amazon SQS as my queue driver. I run an hourly process which pushes a whole bunch of jobs to the queue.
The problem I was experiencing was that there were too many jobs pushed to the queue and my single worker dyno was only processing 1 message at a time, which meant by the time the next cycle came, more jobs was being pushed and my worker dyno was falling behind trying to catch up and process the jobs.
I was looking for a way to process more jobs in the queue concurrently and it seemed possible after reading up on Laravel’s documentation on Supervisor. Supervisor is a great way to manage your queue workers, specifying the number of instances to run and also restarting your queue worker if it fails.
In my case, I wanted to have more workers working on my queue to process the jobs quicker, but the only want to do so on Heroku was to either scale horizontally or to create more processes in your Procfile (free and hobby dynos have a limit of 1 dyno per process type, so you can’t scale).
This seemed like a waste of resources for me since my dynos had a limit of 512MB RAM and I was only utilizing like < 64MB consistently. I figured by increasing the number of workers working concurrently, I could utilize more RAM before needing to scale more dynos.
The Solution
So after hours and days spent Googling for a solution, it was very surprising that there are no documentations out there on how to run Supervisor on Heroku. After lots of tinkering around, I think I may have figured out how to do it.
Heroku Buildpacks
Since I’m running Laravel, my buildpack is heroku/php. However, you will need the heroku/python buildpack to run Supervisor since it is written in python.
To add an additional buildpack, using Heroku’s CLI, run
heroku buildpacks:add --index 1 heroku/python
This will set the heroku/python buildpack in position 1 and the heroku/php buildpack in position 2. The order of this is important. You can also run heroku buildpacks to verify this.
Python Dependencies & Version
Create a requirements.txt file in your app’s root directory and add supervisor. This will install Supervisor when your app is being built after you push a commit to Heroku.
According to Heroku’s documentation, newly created Python applications will run version 3.6.2 but Supervisor will only work with Python 2.4 or later and will not working with Python 3+. This means you will need to change your runtime to version 2.7.13.
Create a runtime.txt file in your app’s root directory and add python-2.7.13. When you push to production later, you should see the following during the build to verify Supervisor is being installed with the correct version of Python.
remote: -----> Python app detected remote: -----> Installing python-2.7.13 remote: -----> Installing pip remote: -----> Installing requirements with pip remote: Collecting supervisor (from -r /tmp/build_a86d7c5833de0290e4d9eae6bb15574d/requirements.txt (line 1)) remote: Downloading supervisor-3.3.3.tar.gz (418kB) remote: Collecting meld3>=0.6.5 (from supervisor->-r /tmp/build_a86d7c5833de0290e4d9eae6bb15574d/requirements.txt (line 1)) remote: Downloading meld3-1.0.2-py2.py3-none-any.whl remote: Installing collected packages: meld3, supervisor remote: Running setup.py install for supervisor: started remote: Running setup.py install for supervisor: finished with status 'done' remote: Successfully installed meld3-1.0.2 supervisor-3.3.3
Supervisor Configuration for Laravel
You can follow the instructions on Laravel to create a config file for your worker. These are my configurations for my app specifically,
[program:laravel-worker] process_name=%(program_name)s_%(process_num)02d command=php /app/artisan queue:work --queue=queue_name --tries=3 --memory=64 --sleep=3 autostart=true autorestart=true numprocs=8 redirect_stderr=true stdout_logfile=/app/worker.log
I’m running my queue worker with a limit of 64MB RAM and running 8 instances to maximize my 512MB RAM allocation (64MB x 8 = 512MB).
Make sure your path to artisan is correctly set. Assume I’ve saved this to laravel-worker.conf.
Supervisor Configuration
After adding the supervisor config file, python buildpack, dependencies and runtime version, commit the files and deploy to Heroku. Once deployed, run heroku run bash to gain access to your dyno.
Once inside your dyno’s terminal, run echo_supervisord_conf which will print out a sample configuration. Copy and paste this into a file and save it locally (let’s assume supervisor.conf). Heroku runs an ephemeral filesystem which means no files created during runtime are saved in the filesystem once the dyno stops or restarts (hence why you need to save locally first and commit later).
In the [include] section, uncomment and add,
[include] files = laravel-worker.conf ; update appropriately to where your Laravel config file is
This will tell Supervisor to include other configuration files within the configuration. Make sure to assign the path of your config file relative to where the supervisor.conf file is.
In the [supervisorctl] section, the serverurl will be set to something like unix:///tmp/supervisor.sock. You will need to update this as this socket is only accessible to root. Change it to,
[supervisorctl] serverurl=http://127.0.0.1:9001
Otherwise, you’ll run into this error,
error: <class 'socket.error'>, [Errno 111] Connection refused: file: /app/.heroku/python/lib/python2.7/socket.py line: 575
Putting It All Together
Lastly we need tell Heroku to run this as a process by updating the Procfile. Add the following line,
supervisor: supervisord -c supervisor.conf -n # update config path relative to Procfile
Make sure to include the -n flag to run supervisord in the foreground, otherwise it will crash (I’m not sure why). Commit all changes and push to Heroku again.
Checking the logs (run heroku logs --tail) you should see Supervisor initialized, your Laravel config file parsed and your workers being spawned.
INFO Included extra file "/app/supervisor/laravel-worker.conf" during parsing INFO RPC interface 'supervisor' initialized CRIT Server 'unix_http_server' running without any HTTP authentication checking INFO supervisord started with pid 17 INFO spawned: 'laravel-worker_00' with pid 20 INFO spawned: 'laravel-worker_01' with pid 21 INFO spawned: 'laravel-worker_02' with pid 22 INFO spawned: 'laravel-worker_03' with pid 23 INFO spawned: 'laravel-worker_04' with pid 24 INFO spawned: 'laravel-worker_05' with pid 25 INFO spawned: 'laravel-worker_06' with pid 26 INFO spawned: 'laravel-worker_07' with pid 27 INFO success: laravel-worker_00 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) INFO success: laravel-worker_01 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) INFO success: laravel-worker_02 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) INFO success: laravel-worker_03 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) INFO success: laravel-worker_04 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) INFO success: laravel-worker_05 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) INFO success: laravel-worker_06 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) INFO success: laravel-worker_07 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
I can also see my 8 spawned workers processing my SQS queue 8 messages at a time.
Finally, I can also see my dyno’s RAM being utilized more now.
Now I have 8 queue workers working against my queue using a single dyno, rather than having to use 8 separate dynos which would’ve increased my cost significantly.
Known Issues
When running supervisorctl, I’m still running into socket errors as I’m not sure how to bypass the permissions issue on Heroku.
Usually this is needed to run reread and update but since Supervisor starts from scratch every time the dyno boots up, it isn’t necessary as supervisord -c <config file> will include the extra config file during start up.
“When running supervisorctl, I’m still running into socket errors as I’m not sure how to bypass the permissions issue on Heroku.”
Look here – https://stackoverflow.com/questions/18859063/supervisor-socket-error-issue/41470666#41470666
You need to use custom path parameter too.
I like your work. It was really informative and helpful. I was able to set up supervisor successfully on heroku, thanks to you!
And yeah, some parts in this guide can be updated like:
1. Now it’s not necessary to run python 2. Supervisor can be run with python 3 (the default one) by including the updated version of supervisor from its repo by inserting this line in the requirements.txt:
git+https://github.com/Supervisor/supervisor
2. Sometimes the echo_supervisord_conf file is absent for some reason and becomes a trouble to print the sample configuration. So I’m attaching this sample here.
; Sample supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html
;
; Notes:
; – Shell expansion (“~” or “$HOME”) is not supported. Environment
; variables can be expanded using this syntax: “%(ENV_HOME)s”.
; – Quotes around values are not supported, except in the case of
; the environment= options as shown below.
; – Comments must have a leading space: “a=b ;comment” not “a=b;comment”.
; – Command will be truncated if it looks like a config file comment, e.g.
; “command=bash -c ‘foo ; bar'” will truncate to “command=bash -c ‘foo “.
[unix_http_server]
file=/tmp/supervisor.sock ; the path to the socket file
;chmod=0700 ; socket file mode (default 0700)
;chown=nobody:nogroup ; socket file uid:gid owner
;username=user ; default is no username (open server)
;password=123 ; default is no password (open server)
[inet_http_server] ; inet (TCP) server disabled by default
port=*:9001 ; ip_address:port specifier, *:port for all iface
;username=user ; default is no username (open server)
;password=123 ; default is no password (open server)
[supervisord]
logfile=/app/supervisord.log ; main log file; default $CWD/supervisord.log
logfile_maxbytes=50MB ; max main logfile bytes b4 rotation; default 50MB
logfile_backups=10 ; # of main logfile backups; 0 means none, default 10
loglevel=info ; log level; default info; others: debug,warn,trace
pidfile=/tmp/supervisord.pid ; supervisord pidfile; default supervisord.pid
nodaemon=false ; start in foreground if true; default false
minfds=1024 ; min. avail startup file descriptors; default 1024
minprocs=200 ; min. avail process descriptors;default 200
;umask=022 ; process file creation umask; default 022
;user=chrism ; default is current user, required if root
;identifier=supervisor ; supervisord identifier, default is ‘supervisor’
;directory=/tmp ; default is not to cd during start
;nocleanup=true ; don’t clean up tempfiles at start; default false
;childlogdir=/tmp ; ‘AUTO’ child log dir, default $TEMP
;environment=KEY=”value” ; key value pairs to add to environment
;strip_ansi=false ; strip ansi escape codes in logs; def. false
; The rpcinterface:supervisor section must remain in the config file for
; RPC (supervisorctl/web interface) to work. Additional interfaces may be
; added by defining them in separate [rpcinterface:x] sections.
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
; The supervisorctl section configures how supervisorctl will connect to
; supervisord. configure it match the settings in either the unix_http_server
; or inet_http_server section.
[supervisorctl]
;serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
serverurl=127.0.0.1:9001 ; use an http:// url to specify an inet socket
;username=chris ; should be same as in [*_http_server] if set
;password=123 ; should be same as in [*_http_server] if set
;prompt=mysupervisor ; cmd line prompt (default “supervisor”)
;history_file=~/.sc_history ; use readline history if available
; The sample program section below shows all possible program subsection values.
; Create one or more ‘real’ program: sections to be able to control them under
; supervisor.
;[program:theprogramname]
;command=/bin/cat ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1 ; number of processes copies to start (def 1)
;directory=/tmp ; directory to cwd to before exec (def no cwd)
;umask=022 ; umask for process (default None)
;priority=999 ; the relative start priority (default 999)
;autostart=true ; start at supervisord start (default: true)
;startsecs=1 ; # of secs prog must stay up to be running (def. 1)
;startretries=3 ; max # of serial start failures when starting (default 3)
;autorestart=unexpected ; when to restart if exited after running (def: unexpected)
;exitcodes=0 ; ‘expected’ exit codes used with autorestart (default 0)
;stopsignal=QUIT ; signal used to kill process (default TERM)
;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
;stopasgroup=false ; send stop signal to the UNIX process group (default false)
;killasgroup=false ; SIGKILL the UNIX process group (def false)
;user=chrism ; setuid to this UNIX account to run the program
;redirect_stderr=true ; redirect proc stderr to stdout (default false)
;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10 ; # of stdout logfile backups (0 means none, default 10)
;stdout_capture_maxbytes=1MB ; number of bytes in ‘capturemode’ (default 0)
;stdout_events_enabled=false ; emit events on stdout writes (default false)
;stdout_syslog=false ; send stdout to syslog with process name (default false)
;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups=10 ; # of stderr logfile backups (0 means none, default 10)
;stderr_capture_maxbytes=1MB ; number of bytes in ‘capturemode’ (default 0)
;stderr_events_enabled=false ; emit events on stderr writes (default false)
;stderr_syslog=false ; send stderr to syslog with process name (default false)
;environment=A=”1″,B=”2″ ; process environment additions (def no adds)
;serverurl=AUTO ; override serverurl computation (childutils)
; The sample eventlistener section below shows all possible eventlistener
; subsection values. Create one or more ‘real’ eventlistener: sections to be
; able to handle event notifications sent by supervisord.
;[eventlistener:theeventlistenername]
;command=/bin/eventlistener ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1 ; number of processes copies to start (def 1)
;events=EVENT ; event notif. types to subscribe to (req’d)
;buffer_size=10 ; event buffer queue size (default 10)
;directory=/tmp ; directory to cwd to before exec (def no cwd)
;umask=022 ; umask for process (default None)
;priority=-1 ; the relative start priority (default -1)
;autostart=true ; start at supervisord start (default: true)
;startsecs=1 ; # of secs prog must stay up to be running (def. 1)
;startretries=3 ; max # of serial start failures when starting (default 3)
;autorestart=unexpected ; autorestart if exited after running (def: unexpected)
;exitcodes=0 ; ‘expected’ exit codes used with autorestart (default 0)
;stopsignal=QUIT ; signal used to kill process (default TERM)
;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
;stopasgroup=false ; send stop signal to the UNIX process group (default false)
;killasgroup=false ; SIGKILL the UNIX process group (def false)
;user=chrism ; setuid to this UNIX account to run the program
;redirect_stderr=false ; redirect_stderr=true is not allowed for eventlisteners
;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10 ; # of stdout logfile backups (0 means none, default 10)
;stdout_events_enabled=false ; emit events on stdout writes (default false)
;stdout_syslog=false ; send stdout to syslog with process name (default false)
;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups=10 ; # of stderr logfile backups (0 means none, default 10)
;stderr_events_enabled=false ; emit events on stderr writes (default false)
;stderr_syslog=false ; send stderr to syslog with process name (default false)
;environment=A=”1″,B=”2″ ; process environment additions
;serverurl=AUTO ; override serverurl computation (childutils)
; The sample group section below shows all possible group values. Create one
; or more ‘real’ group: sections to create “heterogeneous” process groups.
;[group:thegroupname]
;programs=progname1,progname2 ; each refers to ‘x’ in [program:x] definitions
;priority=999 ; the relative start priority (default 999)
; The [include] section can just contain the “files” setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
[include]
files = /app/config/program1.conf /app/config/progra2.conf