|Tweets by @markusgattol||
Make the easy things easy and the hard things possible.
This section gathers basic information about Django.
Why would one be choosing Django? Below is a rough outline of why people favour Django:
Can what happened to Zope2 happen to Django too? Sure, if we ever forget/ignore one plain fact:
Regardless of how smart, creative, and innovative your
We must make things generic enough so Django bits and pieces work for any Python project. That is true the other way around too — we must use ready-made Python bits and pieces and not cook our own soup from scratch if not absolutely necessary. If we do not, Django will become another Zope2, die and be forgotten rather sooner than later.
There is no certain date. There are several porting efforts whereas this one seems to be the most advanced and serious one. One milestone will be dropping Python 2.4 support which is scheduled for Django 1.4.
Update: We might actually see Python 3 support in late summer 2011 already. Hooray!
There are several content management systems like Django CMS or FeinCMS. Then there is Satchmo and Banjo for example. Aside from those well-known and well-established ones, there are more — please go here or use some Internet search engine to get an idea about the current situation yourself.
Folks love super-snappy web applications therefore we shall give them what they want. Firstly, starting with the most obvious one, also creating the least effort — we get better hardware i.e. we might use a RAID array and lots of RAM (Random Access Memory), all spiced up with some crazy-horse server CPU setup. Secondly, after we threw bigger hardware at our snail-problem, we start caching. Thirdly, if all that is still not enough, we hire additional staff in order to set up a world-class clustered Django + MongoDB setup.
Any Django project has four of them — models (the data tier), views (the logic tier), URL-patterns (maps URLs to particular code parts/sections within the logic tier) and templates (the presentation tier).
Well, if installed using
sa@wks:~$ type dpl; dpl python-djan* | grep ii dpl is aliased to `dpkg -l' ii python-django 1.1.1-1 High-level Python web development framework ii python-django-doc 1.1.1-1 High-level Python web development framework ii python-django-extensions 0.4+git200905112140-2 Useful extensions for Django projects sa@wks:~$ ll /usr/share/pyshared/ | grep django drwxr-xr-x 16 root root 4.0K 2009-11-04 07:15 django drwxr-xr-x 11 root root 4.0K 2009-11-12 21:39 django_extensions -rw-r--r-- 1 root root 775 2009-05-16 01:04 django_extensions-0.4.egg-info
We can see the layout on the filesystem using a nifty alias from my
sa@wks:~$ type ta; ta -L 2 /usr/share/pyshared/django ta is aliased to `tree -a -I \.git*\|*\.\~*\|*\.pyc' /usr/share/pyshared/django |-- __init__.py |-- conf | |-- __init__.py | |-- app_template | |-- global_settings.py | |-- locale | |-- project_template | `-- urls |-- contrib | |-- admin | |-- admindocs | |-- auth | |-- comments | |-- contenttypes | |-- csrf | |-- databrowse | |-- flatpages [skipping a lot of lines...] | |-- tree.py | |-- tzinfo.py | |-- version.py | `-- xmlutils.py `-- views |-- debug.py |-- decorators |-- defaults.py |-- generic |-- i18n.py `-- static.py 51 directories, 87 files sa@wks:~$
Note that the layout on the filesystem reflects how we import Python
code i.e. an
sa@wks:~$ grep -A2 "^def resolve" /usr/share/pyshared/django/core/urlresolvers.py def resolve(path, urlconf=None): return get_resolver(urlconf).resolve(path) sa@wks:~$
How big is Django in terms of diskspace needed to install it? dlocate can tell us:
sa@wks:~$ dlocate -du python-django | grep total 16624 total sa@wks:~$ date -u Sun Nov 29 11:18:50 UTC 2009 sa@wks:~$
Around 16.6 MiB these days (November 2009).
It is a large suite of non-core Django functionality i.e. the part of the Django codebase that contains various useful add-ons to the core framework. Go here for more information.
So, can we use Django to create a GIS (Geographic Information System)? The answer is yes! For example, there is GeoDjango which is based on PostGIS — http://djangopeople.net for example makes use of GeoDjango. PostGIS puts a number of spatial datatypes into PostgreSQL, and GeoDjango builds onto that. Go here for more information about Django's storage options.
We can think of
The admin site for example is one part of
The term container is, as of now (November 2010), not part of the official terminology used within the Django community — some use it, many do not. However, I think it is very helpful in approaching and understanding other basic concepts and conventions which are, in fact, official Django concepts/practices and therefore part of the official and everyday Django parlance.
Now, let us take one step back and look at the big picture... Many people, especially when new to Django, ask the most basic questions:
The first two are discussed on other sections/pages on this website but the last one is where we drill down now... first thing we come across is... you guessed it, containers! However, before we start drilling down on what does the filesystem layout for a Django project look like, let us go over a view basic things:
The major guidelines are similar to any other code project. Django applications should address a single, clearly-defined responsibility. The name application is actually a misnomer — Django applications should be thought of more as reusable components which can be plugged together in order to create a web application.
So what we end up with is containers — one outer container (Python/Django project) containing one or more inner containers (Django applications).
The entire outer container e.g. a Django project then makes up for a web application which caters to a particular domain object. The point being that a container, whether outer or inner one, groups together what semantically/logically belongs together.
For example, we might have a web application which domain object is a social networking site (think mini-Facebook). From a technical point of view this web application might be made of a Django project which contains a bunch of Django applications (e.g. email, photo sharing, friends, file storage, etc.) which together make up for this social networking site — or shall we say web application with domain object social networking site ;-]
Anyhow, those would all just be directories on the filesystem, the
outer container being the Django project container, grouping together
all the application containers (subdirectories) which in turn group
together task specific things e.g. the Django application
Below is how it would look like on the filesystem, one root directory
(outer container) with a bunch of subdirectories i.e. one outer
container and one or more inner containers (
sa@sub:/tmp$ pwd; type ta; ta mini-facebook.com/ /tmp ta is aliased to `tree --charset ascii -a -I \.git*\|*\.\~*\|*\.pyc' mini-facebook.com/ |-- email |-- file_storage |-- friends `-- photo_sharing sa@sub:/tmp$
We already know that, from a technical point of view, Django applications as well as Django projects, both are just Python packages. Tests for each Django application should be contained within that particular Django application — Django applications should be decoupled from each other as much as possible so they become reusable, but clearly there will be dependencies, so the goal should be to keep the dependency graph as simple and lightweight as possible.
It is recommended to keep all the templates for a Django project under a single project-wide template directory and/or within subdirectories for each application (using a template subdirectory for each application is a very strong convention in Django, as it avoids template name collisions between applications). The reason for a single project-wide templates directory is that templates, template inheritance trees, and block names can be quite project-specific, so it is hard to provide default application templates that can plug into any Django project.
So what is the takeaway here? Well, the takeaway is that as inner containers (Django projects), very much like outer containers, group together what semantically/logically belongs together e.g. a Django application becomes reusable only if it ships with its own templates, tests, models, views, etc.
Next, extending on the example from above, we are now showing the most
basic filesystem layout for a web application build as a Django-based
project (outer container) with a bunch of Django applications (inner
1 sa@wks:/tmp$ ta mini-facebook.com/ 2 mini-facebook.com/ 3 |-- email 4 | |-- models.py 5 | |-- templates 6 | |-- tests 7 | `-- views.py 8 |-- file_storage 9 | |-- models.py 10 | |-- templates 11 | |-- tests 12 | `-- views.py 13 14 15 [skipping a lot of lines...] 16 17 18 |-- models.py 19 |-- settings.py 20 |-- templates 21 |-- tests 22 `-- views.py 23 24 sa@wks:/tmp$
As we can see, we have project specific/wide (line 20) as well as application specific parts (lines 5 and 10). As mentioned before, we make this distinction so Django applications become reusable across Django projects.
Now, with all the knowledge about containers and naming conventions we backup a bit by saying that all this is more like a recommendation rather than set in stone requirements:
Even with those terms and recommendations/conventions, the key thing to realize is that there is nothing magic about a certain directory or directory structure, we can arrange things however it suits our workflow and make it work.
Understanding the distinction Django draws between a project, an application and a site is mandatory for anybody who wants to do good code layout on the filesystem, write portable software, and most importantly create scalable and long-term maintainable web applications using Django.
It is also worth mentioning that a Django project is not the same as a Python project — the difference is that a Django project is actually what we call package in Python i.e. a Django project lives inside (is a subdirectory of) a Python project.
How we arrange these is really up to our project. In a complicated case, we might do:
Project: ExampleProject App: Web Version Site: example-foo.com Site: example-bar.com App: XML API Version Site: example-foo.com Site: example-bar.com Common non-app settings, libraries, auth, etc
Or, for a simpler project that wants to use one of the many available FLOSS (Free/Libre Open Source Software) add-ons:
Project: ExampleProject App: Example (No specific use of the sites feature... it's just one site) App: Plug-in TinyMCE editor with image upload (No specific use of the sites feature)
Views, custom manipulators, custom context processors and most other things Django lets us create can all be defined either at the project level or the application level. Generally, though, they are best placed inside an application (this increases their portability across projects).
Aside from the fact that there needs to be a project, and at least one application, the arrangement is very flexible — we can design the filesystem layout to adapt whatever suits us best to help abstract and manage the complexity (or simplicity) of our deployment.
Django provides us with an automatic admin interface also known as admin site. Django does so by reading metadata from our models which it then uses to provide a powerful and production-ready interface that content producers can immediately use to start adding/deleting/altering content to/from/at their website.
Some common examples where having an admin site might be useful are:
Note that the admin site is entirely optional because only certain
types of websites need this functionality. It is disabled per default
i.e. we need to take a few steps in order to activate an admin site.
Usually that means we need to touch our projects
A Django admin site is represented by an
instance of the class AdminSite found at
If we would like to set up our own admin site with custom behavior,
however, we are free to subclass
If it is a remote machine located within some datacenter for example,
we might use SSH (Secure Shell). If it is locally we do not need that
of course. However, what is needed in both cases is for us to create a
new superuser account which can then be used to log in and alter/reset
the password for the original superuser or delete the original
superuser account altogether and use the new one from now on. The
command used is
sa@wks:~/0/django/mysite$ ./manage.py help createsuperuser Usage: manage.py createsuperuser [options] Used to create a superuser. Options: -v VERBOSITY, --verbosity=VERBOSITY Verbosity level; 0=minimal output, 1=normal output, 2=all output --settings=SETTINGS The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the DJANGO_SETTINGS_MODULE environment variable will be used. --pythonpath=PYTHONPATH A directory to add to the Python path, e.g. "/home/djangoprojects/myproject". --traceback Print traceback on exception --username=USERNAME Specifies the username for the superuser. --email=EMAIL Specifies the email address for the superuser. --noinput Tells Django to NOT prompt the user for input of any kind. You must use --username and --email with --noinput, and superusers created with --noinput will not be able to log in until they're given a valid password. --version show program's version number and exit -h, --help show this help message and exit sa@wks:~/0/django/mysite$
We now know about the basics. Next we are going to take a first glance at what it means to develop Django-based web applications. After this subsection we will be familiar with all core concepts in Django.
This one is about connecting the dots — where are things located on the filesystem and how do they make their way down the pipe to the users web browser that is.
Django # introduces MEDIA_ROOT, MEDIA_URL, STATIC_ROOT, STATIC_URL, STATICFILES_DIRS Python # introduces PROJECT_ROOT Operating System # introduces WORKON_HOME
As we can see, since the stack grows bottom to top, a Django project
would use/have all of the environment/Python variables. A
Python project would only have
Now, assuming that we have a Django project, we also need to know about
The latter three are part of the puzzle in case we are using django-staticfiles which we do automatically these days (November 2010) because it went into Django with version 1.3. Generally, when discussing where things are located on the filesystem, there are three main areas of concern:
After the introduction from above, we can now have a closer look at those particular environment/Python variables.
In short: A model is the software layer (code to store/retrieve/alter/etc.) atop the data and the data itself.
Models are used to execute SQL code behind the scenes and return convenient Python data structures representing the rows in our database tables.
A Django model is a description of the data inside the database,
represented as Python code. It is our data layout i.e. the equivalent
Models are also used to represent higher-level concepts that SQL cannot handle like for example functionality for a particular model. In other words: A Django model not just describes the database table layout for an object but it also describes any functionality an object knows about itself.
Let us take
Model metadata is anything that is not a field such as the use of
Meta options are used within
No model metadata is required, and adding it to a model is completely optional.
We can add methods to a model in order to get custom row-level functionality (i.e. per single instance of a model instance/object) for our objects. Model methods act on object instances whereas manager methods on the other hand are intended to do table-wide things.
Adding model methods to a model is a valuable technique for keeping business logic in one place — the model itself that is.
A model's manager is a Python object through which Django models
perform database queries i.e. an interface for database queries in
Django. Each model object/instance has at least one manager object
A manager is used any time we want to look up model instances i.e. managers take care of all table-level operations on data including, most importantly, data lookup.
Managers are accessible only via model classes, rather than from model instances, to enforce a separation between table-level operations and record-level operations.
Adding extra model manager methods is the preferred way to add table-level functionality to our models i.e. functions that act on all instances of a models instances/objects. For row-level functionality i.e. functions that act on a single instance of a model instance/object, using model methods is the way to go.
A queryset represents a collection of objects from our database(s) returned by a query. It can have filters i.e. criteria that narrow down the collection of objects based on given parameters.
In SQL terms, a queryset equates to a
A queryset is an object itself. It is constructed via a Manager on
some model class. For example if we had a model called
As can be seen, querysets in its simplest form provide us with an easy and efficient way to execute all kinds of queries on our data. Using filters makes things a lot more versatile and easy — in 9 out of 10 cases, that is all we ever need. However, if filters are still not enough to get the job done, querysets provide us with the ability of using so called F or Q objects.
Note that lookup functions (such as
Querysets can be cached, effectively boosting application speed when used correctly i.e. the database(s) are only queried once if asked for the same queryset more than once.
We use models to store/retrieve/alter information. However, there is not just information inside each model, but also in the relationships amongst them...
The whole is more than the sum of its parts.
Please go here for more information.
Both are model field options. They are optional. Django uses default
class Author(models.Model): first_name = models.CharField(max_length=30) last_name = models.CharField(max_length=40) email = models.EmailField()
Every author has a first name and a last name but not necessarily an
E-mail address. The above model however requires us to provide an
E-mail address to every author. We can make it so that providing an
E-mail address becomes optional if we use
Well, SQL has its own way of specifying blank values — a special
To help avoid such ambiguity, Django's automatically generated
CREATE TABLE "books_author" ( "id" serial NOT NULL PRIMARY KEY, "first_name" varchar(30) NOT NULL, "last_name" varchar(40) NOT NULL, "email" varchar(75) NOT NULL )
Excellent! We have all SQL columns set to
In Django models, we can specify that a database column SQL
We change our
class Author(models.Model): first_name = models.CharField(max_length=30) last_name = models.CharField(max_length=40) email = models.EmailField(blank=True) author_added_to_database = models.DateTimeField(blank=True, null=True)
CREATE TABLE "books_author" ( "id" serial NOT NULL PRIMARY KEY, "first_name" varchar(30) NOT NULL, "last_name" varchar(40) NOT NULL, "email" varchar(75) NOT NULL "author_added_to_database" timestamp with time zone )
Not so with shortcuts — those basically allow us to span multiple MVC
layers e.g. grab an HttpRequest object and use the
A template is used to give structure to our user-facing part of the web application. It is a simple text file used to generate any text-based output format e.g. HTML, XML, etc. The output of templates together with CSS makes for the so-called look and feel of our web application i.e. the way it appears to the user.
Both, structure and presentation, live within the presentation tier of our web application — rather than within the data (models) or logic (views) tier.
Internally, a Django template is represented as a collection of nodes.
Nodes are Python classes which all inherit from the base node class
Nodes can do various sorts of processing, but they have one thing in
common: every Node must have a method called
A context is a variable name to variable value mapping that is passed to a template. The data structure used for this mapping is a Python dictionary, mapping template variable names (dictionary keys) to Python objects (dictionary values).
A template then renders a context by replacing the variable holes with values from the context and executing all block tags.
It is a Python module called URLconf containing mappings between URL (Uniform Resource Locator) patterns (regular expressions) and views i.e. it decouples URLs from Python code (the logic-tier in Django also known as views; see MTV).
Note that these mappings do not search
A form is a collection of fields that knows how to validate itself and display itself as HTML. Django has a dedicated form library which allows us to do a bunch of things:
The library is decoupled from other Django components, such as the
database layer, views and templates. It relies only on Django
settings, a couple of
The distinction between bound and unbound forms is important. An unbound form instance does not have any data associated with it i.e. when rendered to the user, it will be empty or will contain default values. Of course, because it has no data associated with it, there is nothing to validate!
A bound form instance does have submitted data associated with it
(from a POST), and hence can be used to tell if that data is valid
If our form is going to be used to directly add or edit a Django model, we can use a ModelForm to avoid duplicating our model description.
A view has to be a callable. It usually is a Python callback function
that does something (or not as in
The rest is up to us i.e. our view can (or not) read/write from/to a
database. It can (or not) use a template system, either Django's
built-in template system or a third-party template system. A view
might also generate a PDF file, output XML, create a ZIP file on the
fly, anything we want, using whatever Python libraries we want, there
is no built-in limitation to what a view can do except, Django either
A generic view is no different except it is a higher-order view that provides an abstract/generic implementation of a common idiom or pattern found in view development i.e. a generic view is a ready-made view we can use without the need to write a view ourselves. In other words: Django provides us with a bunch of views for common/recurring cases so we do not have to code them over and over again.
The two most popular generic views are
We have tons of them. In Django parlance those extensions are called Django applications.
First let us clarify on the matter: There is core Django and then there are hundreds if not thousands of additional extensions, written by third parties, that can be used to extend Django's functionality and/or change its core behavior somehow.
Note that code that makes up our Django project (code that builds on Django and creates some added value which ultimately ends in being an individual project i.e. what users visit using their web browser) is not necessarily what we call a Django extension.
Only if code which builds upon Django (or portions of it) can be reused in other Django projects as well do we recognize it as an extension. All the rest that cannot be reused is considered code that makes our project unique.
Of course, every project ultimately has some portions of code that either cannot be reused or of which reusing does not make any sense. The point here is, the more code can be reused outside our project, the better it is. Into that... Pinax is an effort towards the goal of maximizing code reuse and thus minimizing repeating tasks and code redundancy. This is a core principle called DRY (Don't repeat yourself) which, for good reasons, is very prominent amongst Python/Django developers.
To answer the question about available extensions: There are so many and changes often happen frequently that listing them here does not make any sense. The right place to look for extensions is the PyPI (Python Project Index), where extensions can be shared and explained with/to others.
In Django parlance those extensions are called Django applications. Here is information about how to build and maintain Django reusable apps.
Please go here.
We now know about all the basic concepts of Django and all the parts that resemble a Django-based web application, we are now going to have a look at a few basic guidelines which are considered best practice and should thus be followed.
Depends. Here are a bunch of links with different angles on different things:
We now know what it takes to create a Django-based web application. This subsection is about tools and procedures which help us boost productivity, ease the overall process of software development, and last but not least, help us make sure that our web application is fast/efficient and has the lowest bug-count possible.
sa@wks:/usr/share/pyshared/django_extensions/db$ grep 'models.Model' models.py class TimeStampedModel(models.Model): class TitleSlugDescriptionModel(models.Model): class ActivatorModel(models.Model): sa@wks:/usr/share/pyshared/django_extensions/db$
sa@wks:/usr/share/pyshared/django_extensions/db/fields$ grep 'class ' __init__.py class AutoSlugField(SlugField): class CreationDateTimeField(DateTimeField): class ModificationDateTimeField(CreationDateTimeField): class UUIDVersionError(Exception): class UUIDField(CharField): sa@wks:/usr/share/pyshared/django_extensions/db/fields$
sa@wks:~/0/django/mysite$ ./manage.py show_urls | column -t | grep -v /admin /rosetta/ rosetta.views.home rosetta-home /rosetta/pick/ rosetta.views.list_languages rosetta-pick-file /rosetta/download/ rosetta.views.download_file rosetta-download-file /rosetta/select/<langid>/<idx>/ rosetta.views.lang_sel rosetta-language-selection sa@wks:~/0/django/mysite$
This subsection is about knowing what is going on with our setup i.e. if it is healthy or not, if we need to add more servers to cope with growth etc.
This subsection is about money when it is shoot back and forth as ones and zeros across wires, stored onto non-volatile memory and processed by random computing devices.
This subsection is about scaling our Django-based web application i.e. how we architecture it (the entire stack i.e. presentation tier, logic tier and data tier) so we can keep up with a growing number of requests and a growing data set.
A scalable system should not need to undergo fundamental changes in
We can scale in two dimensions, each one being mutually exclusive within a single tier, but possible to combine across tiers:
As indicated already, even if the above examples assume that we choose one approach for the entire stack, note that we can mix both approaches e.g. the data tier might be designed to scale horizontally whereas the logic tier within the same stack might be designed to scale vertically. So, what is better, scaling horizontally or scaling vertically?
Vertically is better if we have plenty of financial resources right from the start so we can afford to buy big machines and the service contracts that go with them but if we are limited in time (as in TTM (Time to Market) for example) and human resources/knowledge to architecture, implement and maintain a horizontally scalable solution.
The downside with the vertical approach however is that it has a practically relevant upper boundary, which, when reached, will make it impossible to grown any further (i.e. if the already fastest/biggest machine can not handle our requests and/or data set anymore). However, as already mentioned, until we arrive at this upper boundary, the good thing about the vertical approach is that the entire development and maintenance cycle stays the same (read easy/low) at all times.
Anyhow, considering that the whole idea of scaling is to not have a practically relevant upper boundary, the vertical approach seems wrong to even begin with — I would even go as far as to say that scaling vertically is a contradiction in itself as we certainly cannot achieve what we can achieve with the horizontal approach which allows us to practically scale infinitely.
Note the usage of the word practically here. It basically determines that we can manage to stay below the upper boundary which we can define as the currently technically possible.
Both, our current needs in terms of how much requests we need to handle and/or the size of the data set we need to store, as well as what is currently considered the currently technically possible upper boundary in software and hardware, they are both moving targets. As long as we manage to keep our needs for speed and size below that technically possible upper boundary, its fair to say that we can practically scale infinitely. That is something which is simply not possible with the vertical approach where it is not unusual that the need for speed and size surpasses the upper boundary for what is the currently technically possible (see example where the already fastest/biggest server becomes to slow/small).
So, assuming there is enough time and human resources as well as just about enough money to start with a setup of commodity hardware at our disposal, scaling horizontally is certainly what everybody should opt for — even if we had enough money from the very beginning to afford expensive hardware, it is probably smarter in the long run to invest in people rather than hardware, license fees and service contracts.
Everything is possible, as usual it all depends on the use case at hand, the problems we need to solve, the domain object, the available time frame, and things like TCO (Total Cost of Ownership) and RoI (Return on Investment), switching costs and if the vendor lock-in is bearable, as well as things which are rather hard to quantify such as how to buildup assets, sustain/gain competitor ship and, most importantly, have fun at what we do.
Distribute incoming requests across many machines...
Not every incoming request should hit the data tier...
Prioritize tasks. What can wait and should be handled asynchronously (maybe even by another machine)?
Scaling the data tier horizontally means distributing the data set as well as load across many servers and not loose data (i.e. also have it replicated).
This subsection is considered to be an advanced section i.e. functionality, concepts and tasks which we usually do not need in small to mid-sized web applications and/or features respectively use cases which are not considered to be mainstream so far.
federation, login across several domains
With this subsection we are going to look at how to deploy Django-based web application and how to apply changes/upgrades once they are deployed.
The level of flexibility that can be achieved with the Django CMS 2.0
This section gathers FAQs about Django CMS.
If we take a look at the source code from current HEAD, we can see the
Just some rough notes so far....
Required Debian packages:
Miscellaneous Info or stuff to install/setup:
For information about developing a plugin, please go here.