Latest Blog Posts

Before I get started, I'd like to point out that the code i'm going to demonstrate was inspired by Eric Florenzano's post Tagging Cache Keys for O(1) Batch Invalidation.

The concept is pretty simple. Everyone knows that the hardest thing about working with a cache is dealing with invalidation. when something in the database is changed, you want to clear the old invalid data out of the cache. but to do that, you need to know exactly which cache keys need to be removed. What's worse, if you have thousands of dynamically generated keys. Not only is it difficult to track them all, but the invalidation could take quite some time too.

The solution to both of the problem is quite brilliant really, and I want to again thank Eric Florenzano for his post which inspired this one. go check out his blog, its filled with great django tips.. or course, finish reading this first, What we're going to do is explain how the technique works, and then we're going to wrap it up in a nice module that you can import in place of django.core.cache.cache and use it just as you always would, but with the added benefit of this awesome key-grouping technique.

The "Trick" here, is that we aren't going to actually delete anything from the cache. We're just going to let it be purged when the cache fills up. Instead of deleting data, we need to alter the cache keys going into all of the standard cache functions and insert a small code that is unique to the group that you want this key to belong to. and then when you want to invalidate those entries, you simply change that unique code for the group, and all of a sudden the cache keys are different and the lookups will fail.

We're going to store the mapping of these group names to codes directly in the cache so that we can simply delete or overwrite that single key for a group in order to invalidate the whole set of related keys. We're also going to use uuid's, because that's what eric's example used, and they're guaranteed unique, and easy to generate.

The interface I'm going to offer is is just a set of wrappers around the standard django cache functions which all accept a single 'group' keyword argument.

from django.core.cache import cache

def get(key, default=None, group=None):
    if group:
        key = _make_key(group, key)
    return cache.get(key, default=default)

def set(key, value, timeout=0, group=None):
    if group:
        key = _make_key(group, key)
    return cache.set(key, value, timeout=timeout)

def add(key, value, timeout=0, group=None):
    if group:
        key = _make_key(group, key)
    return cache.add(key, value, timeout=timeout)

def delete(key, group=None):
    if group:
        key = _make_key(group, key)
    return cache.delete(key)

def get_many(keys, group=None):
    hashkey = _get_hashkey(group)
    keys = [_make_key(group, k, hashkey) for k in keys]
    return cache.get_many(keys)

def incr(key, delta=1, group=None):
    if group:
        key = _make_key(group, key)
    return cache.incr(key, delta=delta)

def decr(key, delta=1, group=None):
    if group:
        key = _make_key(group,  key)
    return cache.decr(key, delta=delta)

as you can see here, we dont do anything at all unless the group keyword argument is pass in, that allows this module to swapped in place of the standard django cache with no problems. if the group argument is passed into one of these functions, we use it to look up or generate a new UUID for the group.. the function which does this is defined as such:

from django.conf import settings
import uuid

# This prefix is appended to the group name to prevent cache key clashes.
_KEY_PREFIX = getattr(settings, 'BCACHE_KEY_PREFIX', "bcache__")

def _get_hashkey(group_name):
    hashkey = cache.get("%s%s" % (_KEY_PREFIX, group_name), None)
    if not hashkey:
        hashkey = uuid.uuid4()
        cache.set("%s%s" % (_KEY_PREFIX, group_name), hashkey)
    return hashkey

def _make_key(group_name, cache_key, hashkey=None):
    """ 
    Generates a new cache key which belongs to a group 
    
    """
    # This can be useful sometimes if you're doing a very large number 
    # of operations and you want to avoid all of the extra cache hits.
    if not hashkey:
        hashkey = _get_hashkey(group_name)
    return "%s__%s-%s" % (group_name, cache_key, hashkey)

We also need a convenient way to invalidate groups... so we have this function:

def invalidate_group(group_name):
    """
    Invalidates all cache keys belonging to group_name
    
    """
    cache.delete("%s%s" % (_KEY_PREFIX, group_name))

Pretty simple eh? Thats all there is to it. To demo our new cache group functions, here is an example view which caches results for various search terms.

def search(request):
    query = request.GET.get('q','')
    cache_key = "results-for-%s" % query
    results = cache.get(cache_key, group='search-results')
    if not results:
        results = Thing.objects.filter(text__icontains=query)
        cache.set(cache_key, results, 999999, group='search-results')        

Then maybe in our model's custom save handler, we might want to invalidate the whole group when something changes.

class Thing(models.Model):
    name = models.CharField(max_length=255)
    text = models.TextField()
    def save(self, *args, **kwargs):
        cache.invalidate_group('search-results')
        super(Thing, self).save(*args, **kwargs)

This has saved me a lot of hassle. if offers a nice clean interface that is just as easy to use as regular cache functions, which is great. as always, suggestions and improvements are appreciated.

This post will show you how to extend the django admin to allow for any model to be exported to an excel spreadsheet. it uses the xlwt module to generate the spreadsheet.

views.py

def admin_export_xls(request, app, model):
    mc = ContentType.objects.get(app_label=app, model=model).model_class()
    wb = xlwt.Workbook()
    ws = wb.add_sheet(unicode(mc._meta.verbose_name_plural))
    for i, f in enumerate(mc._meta.fields):
        ws.write(0,i, f.name)
    qs = mc.objects.all()
    for ri, row in enumerate(qs):
        for ci, f in enumerate(mc._meta.fields):
            ws.write(ri+1, ci, unicode(getattr(row, f.name)))
    fd, fn = tempfile.mkstemp()
    os.close(fd)
    wb.save(fn)
    fh = open(fn, 'rb')
    resp = fh.read()
    fh.close()
    response = HttpResponse(resp, mimetype='application/ms-excel')
    response['Content-Disposition'] = 'attachment; filename=%s.xls' % \
          (unicode(mc._meta.verbose_name_plural),)
    return response

because the xlwt module doesn't allow you to write to an open file handle, we have to first create a temporary file using tempfile.mkstemp() and subsequently close the file, then save spreadsheet with the returned filename. This might not be the most efficient way, but it works.

in the urls file we need to point to the new view, be sure to put this line before the regular admin include since it acts as a sort of catch-all. [Update: Wrapped the view in admin.site.admin_view decorator.] urls.py

...

    (r'^admin/([^\/]+)/([^\/]+)/xls/$', admin_export_xls),
    (r'^admin/(.*)', admin.site.admin_view(admin.site.root)),
...

Now we need to add a button to the top of each change list template that we want to have this functionality.

change_list.html

{% extends 'admin/change_list.html' %}{% load i18n %}
{% block object-tools %}
<ul class="object-tools">
  {% if has_add_permission %}
  <li>
<a href="add/{% if is_popup %}?_popup=1{% endif %}" class="addlink">
      {% blocktrans with cl.opts.verbose_name as name %}Add {{ name }}{% endblocktrans %}
    </a>
</li>
  {% endif %}
  <li>
<a href="xls/" class="viewsitelink">Export to .XLS</a>
</li>
</ul>
{% endblock %}

Comments/suggestions appreciated.

So, I was coding a django site which required displaying random ads on a page, each ad is given a weight which determines the frequency at which the ad is displayed. I will admit that my first thought was far from optimal, but I later came across a post on stackoverflow.com, with a more appropriate algorithm. The code on stackoverflow was written in C#, which is just... ugly, so here is the python version.

def random_by_weight(items, attr='weight'):
    """
    Randomly selects one item from 'items', using the
    'attr' property as a weight field.

    """
    
    total_weight = sum((getattr(i, attr) for i in items))
    rweight = random.randint(0, total_weight)
    for item in items:
        if rweight <= getattr(item, attr):
            return item
        rweight -= getattr(item, attr)

...and there you have it.