How to install Celery on Django and create a periodic task

Celery 3.1.19 and Django 1.8.7.
This post explains how to set up Celery with Django, using RabbitMQ as a message broker.

It also explains how to create a Periodic Task.

The Broker RabbitMQ

First, we need to choose what is called a Message Broker, required by Celery in order to send and receive messages. Here we will use RabbitMQ, which is feature-complete, stable, durable and easy to install.
Moreover, it is the default broker so is does not require additional configuration.

Check out how to install it for your particular system here. If you are using Ubuntu you can install it by following:

1
$ sudo apt-get install rabbitmq

The RabbitMQ server scripts are installed into /usr/local/sbin. This is not automatically added to your path, so open or edit a .bash_profile in your home folder and the following line.

1
export PATH=$PATH:/usr/local/sbin

The server can then be statted with

1
$ sudo rabbitmqctl stop

You can find a detailed description on how to use RabbitMQ with Celery here.

After installing RabbitMQ we need to create a RabbitMQ user, a virtual host and allow that user to access the virtaul host. We also start the server before that:

1
2
3
4
$ sudo rabbitmq-server -detached
$ sudo rabbitmqctl add_user myuser mypassword
$ sudo rabbitmqctl add_host myvhost
$ sudo rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

Then, open your Django project settings.py file and configure RabbitMQ by adding the lines:

1
BROKER_URL = "amqp://myuser:mypassword@localhost:5672/myvhost"

This tells celery, where your broker (your queue) is located. Here, we are running Celery at the same machine as RabbitMQ and using the localhost to find it.

Celery

Celery is on the Python Package Index(PyPi), and can be easily installed with pip or easy_install.

Remember to active first your virtual environment(if you want to install Virtualenv to create a virtual enviroment check this post).

1
$ pip install celery

Next, add this package to your requirements.txt file. so that both the production environment and the development environment on your local machine will use it. Recall that you can check the packages used by the current environment with

1
$ pip freeze

You will see that you have installed celery, pytz, billard, kombu, anyjson and amqp. Write them all on you requirements file. You can also write them directly by using

1
$ pip freeze > requirements.txt

Now, we need to create a Celery instance, called a Celery app. Create a file at the same level of your settings.py file:

1
$ touch myprojectfolder/myproject/celery.py

And write the following code:

1
2
3
4
5
6
7
8
9
10
import os
from celery import Celery
from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE','myproject.settings')

app = Celery('myproject')
app.config_fron_object('django.conf:settings')

app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

Then, to ensure that the app is loaded when Django starts, you need to import this app in the init.py file.

Open the init.py file that is at the same level than the settings.py and celery.py file and write:

1
from .celery import app as celery_app

Moreover, for security purposes, you should specigy a list of accepted content-types in the settings.py file.

In this case, we will set json as our content type:

1
CELERY_ACCEPT_CONTENT = ['json']

Theb we need to specify the task serializer accodingly:

1
2
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALZIER = 'json'

Finally, we can specify the time zone we are in:

1
CELERY_TIMEZONE = 'Asia/Shanghai'

Note: In Celery 3.0+ the setting CELERY_ENABLE_UTC is enabled default (it is set to True). This setting, if enabled, makes the dates and times in messages to be converted to use the UTC timezone.

Django-celery

If you want to store task results in the Django database, you will have to install the django-celery packags.

This package defines a result backend to keep track of the state of the tasks. To install it user:

1
$ pip install django-celery

remember to include it in your requirements file, Then add it to your installed apps in your settings file:

1
2
3
4
INSTALLED_APPS(
	...,
	'djangocelery',
)

Next we need to create the corresponding database tables of this app, which can be done with:

1
$ python manage.py migrate djcelery

As we have indicated Celery to use our settings.py file, we can configure Celery to use the django-celery backend by adding this line into the settings.py file:

1
2
CELERY_RESULT_BACKED = 'djcelery.backends.database:DatabaseBackend'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

Create a Periodic Task

One thing you might want to use in your project is a Scraper, for example, whick is an aplication that runs periodicaly at night to update some data for your web site.

Choose or create an application in your django project to include the Scraper. Then, create and edit the file myapp/utils/scrapers.py (note: you must have an empty init.py file inside the utils folder). The scrapers.py file must contain a function that performs that performs your desired operations, like accessing an API and modifying your database.

In this example, we just write:

1
2
def scraper_example(a,b):
	return a + b

Then, create the file myapp/tasks.py and edit it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
from celery.task.schedules import crontab
from celery.decorators import peeriodic_task
from myapp.utils import scrapers
from celery.utils.log import get_task_logger
from datetime import datetime

logger = get_task_logger(__name__)

@periodic_task(run_every=(crontab(hour='*', minute='*', day_of_week='*')))
def scraper_example():
	logger.info("Start task")
	now = datetime.now()
	result = scrapers.scraper_example(now.day,now.minute)
	logger.info("Task finished: result = %i" % result)

Here we have created a periodic task that will run every minute, and that writes into the logger two messages indicating the beginning and the end of the task, and calls our scraper function.

Run it

Now we created the periodic task, but how we can run it? First start your RabbitMQ server:

1
$ sudo rabbitmq-server -detached

Next start a Celery worker

1
$ python manage.py celeryd --verbosity=2 --loglevel=DEBUG

If the installtion is correct, you should see at the top of the text displayed something like

1
2
transport: amqp://myuser@localhost:5673/myvhost
results: djcelery.backends.database:DatabaseBackend

And a list of the application tasks:

1
2
3
4
[tasks]
  . celery.backend_cleanup
  . ......etc
  . myapp.tasks.scraper_example

Next open a new tab and start celerybeat, which will send the registered tasks periodically to RabbitMQ:

1
$ python manage.py celerybeat --verbosity=2 --loglevel=DEBUG

If you go back to the Celery woeker tab, you will see the results of your tasks.

And finally. open another tab and start your django development server:

1
$ python manage.py runserver

Note: Beat needs to store the last run times of the tasks in a local database file, which by default is celerybeat-schedule.db and it’s placed at the same level of your manage.py file. If you are using GIT as version control, you should include this file into your gitignore file.

That’s all.