Django model creation shortcut with kwargs

Let’s say you’ve got this model in Django that you need to create that has a lot of fields. You’ve already mapped out the data, but you think you need to write in each argument in the Model.objects.create method. Thankfully, these methods allow **kwargs to be passed in so you can do this: 

# Remember that kwargs is just a dictionary that's mapping
# arguments to values (in this case model fields to values)
MyModel.objects.create(**some_dict)

# Or if we are updating it
MyModel.objects.update(**some_dict)

# Conveniently we could do something like this,
# assuming your field names are mapped exactly to the model names.
# Sometimes you don't want a model form so this works.
MyModel.objects.create(**request.POST)

# Oh, but I have a foreign key! No prob. 
MyModel.objects.create(
    fk_field = some_object,
    **some_dict
)

Django Gotcha: Duplicate models when using get_or_create

When you’re saving some object to the db, there’s a gotcha with DateTimeFields that drove me nuts. It took me way too long to figure this out, but if you have a DateTimeField with the option auto_now_add, it will supersede a datetime object in a Model.objects.get_or_create call. This means that if you are tying to prevent duplicates based on date (say you’re saving some data from an API that has a timestamp) you will still get duplicates. It’s in the docs, but not the implications when using get_or_create. Check out this example:

# Given this simple model
class Foo(models.Model):
    name = models.CharField(max_length=100)
    date_added = models.DateTimeField(auto_now_add=True)

# This will always be true, even if an instance
# with this name and today's date already exists

bar, created = Foo.objects.get_or_create(
    name = 'Alex', 
    date_added = some_datetime_obj
)

print created
# >> True


# The problem is, auto_now_add does some stuff that 
# makes it uneditable, and fucks up my expectations
# when using it with get_or_create

# Here's the solution
class Foo(models.Model):
    name = models.CharField(max_length=100)
    date_added = models.DateTimeField(default=datetime.today())

bar, created = Foo.objects.get_or_create(
    name = 'Alex', 
    date_added = some_datetime_obj
)

print created
# >> False

Backbone in the wild, lessons learned using backbone.js

This week I finished up a major redesign of Edorati which helps people discover articles with their friends. The reason I ended up going with a client-side MVC framework, Backbonejs, is because a huge piece of the existing web app was long javascript files with tons of jquery event handling. In short, it was a mess and difficult to build on top of. Although I didn’t set out to use backbone for everything, I ended up doing just that because it made modular development of client side code that much better. It made it feel like javascript was as real as the python/django that powers it. 

Getting Your Head Around Client-Side MVC

Coming from a world of server-side templating, models, and views with Django, I had to start thinking of how that translates to backbone. Your backend will mostly be an API that serves data based on the requested URL and your business logic. Using backbone, my django project only serves one template for the whole app. Everything else is handled by backbone. That should give you an idea of how much heavy lifting backbone is able to do. I ended up stripping 70% of the view code I had written in django. 

Oversimplified Explanation of Backbone

Here is my grossly oversimplified mental model of what backbone is. 

Backbone organizes your front-end javascript code into an MVC style framework. There are 5 primary components to backbone:

  • Models: serialized data that describes an instance (similar to a django model without the database instructions)
  • Collections: a list of models (kind of like a queryset)
  • Views: a handler for a section of your app (similar to a django view, but typically more granular)
  • Router: routes a url to the appropriate view using a hashbang or pushState
  • Templates: Sticks you data into html. Not a part of backbone, but you will almost certainly use them. I personally use handlebars, but you can use whatever you want.

Backbone Patterns No One Tells You About

Going through the tutorials and reading other people’s code I’ve picked up on some basic patterns that are used by convention. Some of them were not immediately obvious when I started learning. Others I learned the hard way.

  1. Granular Views - Consider a page that has a list of articles. With django you might create a view that grabs a queryset and outputs a template with a for loop to produce the list. You might then use jquery to handle events like the user editing that instance. With backbone you would set it up like this: A list view creates a collection of articles, for each model in the collection create an article view. Notice that there are two new pieces in there. The collection which is tied to a url which will call your api then automatically create a list of models from the received data and a view for an individual article. It was not obvious to me why in the world you would want a view for an individual article which might be 5 lines of code. The reason is for event handling. You’re basically setting the scope of events that apply to that code. This will help you avoid large, complicated views that try to handle all sorts of things for individual items. In the long run this leads to better separation of the different parts of your app and makes it much more manageable. Seems like more work up front, but it makes it easier in the end. 
  2. Zombies - I wish the documentation was more up front about this. Any non-trivial app typically creates and changes many views within a single session. YOU HAVE TO CLEAN THAT SHIT UP. If you don’t, your beautiful modular code will exhibit bizarre behavior as soon as you create a new view without first removing the old one. Events are delegated to elements. Even though you may have cleared the html, you have not removed the delegated events. Before you instantiate a new view that replaces another one, remove it and unbind all events from it. (this.remove() this.unbind()). It’s really hard to debug this, but the symptoms are usually things like the wrong ID being used, wrong model is changing etc. Zombies. For more info read this.
  3. Requirejs - Non trivial apps will need to be separated into multiple files. The tutorials out there fail in explaining this. You will almost certainly be needing requirejs. Learn require well. Took me an embarrassing amount of time for me to get this.
  4. One view per file - Related to the requirejs tip above, you only put 1 view in per file. Coming from django I just thought I would put a whole bunch of views in one file, but that’s not how it’s done with backbone using require. One view, model, collection, per file. Why? Because using require you need to return a class and it can’t handle you returning multiple classes. 
  5. Initializing an app - Using require, it’s not obvious how you “boot” the whole app. Create a view that will represent your app and create an initialize function to do what you need to start it up. In your main.js (used by require) create an instance of your app and voila, it will initialize. Typically you would initialize the router and any views that must be there no matter what. For Edorati that means the sidebar and router.
  6. Router is your friend - Let your router handle the switching of views. Don’t try to stick the router logic into your views. Need to send people to a profile detail page? Create a route for it and use that url in the template. Don’t create an event handler in your view so that when someone clicks on it it creates a new view. Just use the router for what it does, route things. Doing this also means you’ll have re-usable links that can be used outside of the session. i.e. link that directly takes you to a user’s profile.

I’m sure I’ll think of more, but that’s it for now. Most of that list will not make sense until you dive into backbone creating your app. Tweet me if you have any questions @alexkehayias.

How to make an interactive bookmarklet part 3

In the last two parts of this series we learned about the framework for building bookmarklets and how to use javascript with django templates for optimal maintainability. Now let’s look at how to make interactions work without needing any external javascript libraries. 

Why not include Jquery

My guess is that all you really want are event bindings and for that you don’t need an entire library. Granted, lots of user’s have jquery cached in their browser already, but if not you have to include that which will make your bookmarklet slower. Every library you add will add more “weight” to the overall bookmarklet. Read the next paragraph before you decide you need the whole jquery library or you can make do with 6 lines of pure javascript.

Binding Events to Handlers

So you want to be fancy and make the bookmarklet change when the user clicks on something? Tough shit. Just kidding. The problem with event handling is that making it work cross-browser is tough. Jquery helps make writing js more cross-browser compatible and that’s a huge part of it’s overall value.

Here’s how we can add event handlers that will work across all browsers without Jquery. I can’t find where I originally found this from, but when I do I’ll add an attribution link.

    var bindEvent = function(elem, evt, cb) {
        //see if the addEventListener function exists on the element
        if ( elem.addEventListener ) {
            elem.addEventListener(evt,cb,false);
        //if addEventListener is not present, see if this is an IE browser
        } else if ( elem.attachEvent ) {
            //prefix the event type with "on"
            elem.attachEvent('on' + evt, function(){
                //use call to simulate addEventListener
                cb.call(event.srcElement,event);
            });
        }
    };

Now we have a simple way to make interactions happen within the bookmarklet itself. For example:

    // Remove the bookmarklet from the document assuming b is body and c is the bookmarklet
    var cancelDialog = function(e){
        b.removeChild(c);
    };
    bindEvent(cancelButton,"click", function(e){
        cancelDialog(this);
    });

All that without including the whole jquery library! That will keep your bookmarklet lightweight and fast. 

Submitting a Form

Using our event binding snippet above we can create a simple handler for submitting a form free from cross-domain errors. Those of you familiar with jsonp will see that this is very similar, but done manually and in a simpler way.

    // Assuming we have the following in the script
    var d=document,
    s=d.createElement("scr"+"ipt");
    b.appendChild(s);
    
    // Bind a submit event to a button click
    var submitArticle = function(e){
        // get the values of any form data you need
        // URI encode the variables as GET parameters
        s.src = "http://www.urltoyourservice.com?var1=" + encodeURIComponent(variable1) + "&var2=" + encodeURIComponent(variable2);
    };
    
    bindEvent(submitButton,"click", function(e){
        submitArticle(this);
    });

This will make a GET request to your your server with the parameters of your form. We can now process that request and return a response just like we did to initialize the bookmarklet in the first place. All we need to do is respond back with javascript. You can do this numerous times. All your doing is manipulating the bookmarklet that is already a part of the document. Make sure that you clean up after yourself (see my earlier example of binding a cancel button to removing the bookmarklet from the document). 

There you have it. An interactive bookmarklet that’s fast, maintainable, and lightweight. You can make changes to it at any time since the initialization happens server-side with familiar templates. You can make cross-domain requests without any errors. You can even bind events and handle them all without needing Jquery. 

Enjoy!

How to make an interactive bookmarklet part 2

Let’s talk about how to generate a bookmarklet using server-side templating and pure javascript (no libraries!). In part 1 we discussed the basic framework for making a bookmarklet and performing requests directly from the client’s page. Now we’ll see how to set up the foundation for a bookmarklet UI and make it look good.

Building a Bookmarklet Template in Django

Let’s create a file called bookmarklet.js. In it we are going to add a self executing piece of javascript and leverage django templates to fill in the information we need. In here we are going to build the response using nothing but javascript and send it back to the user who requested the bookmarklet (remember all they did was request it to be initialized, see part 1 for more info). With django templates we can do some pretty handy things so that you don’t need to manually write out the template as a string in javascript. For example:

javascript:(function(){
    var d=document,
    {% comment %}Create the elements we need {% endcomment %}
    c=d.createElement("div"),
    b=d.body,
    l=d.location;
    {% comment %} server_scheme_and_netlog is my own tag which returns the base url of the website{% endcomment %}
    c.id="bookmarkletArticleAlertBox";
    {% comment %}This is where django templates get's awesome{% endcomment %}    
    {% comment %}Make sure the html rendered has no breaks{% endcomment %}    
    c.innerHTML = '{% spaceless %}{% include "bookmarklet_body.html" %}{% endspaceless %}';
    {% comment %}Add the elements to the dom{% endcomment %}
    b.appendChild(t);
    b.appendChild(c);

....snip....

What we did was construct a div that will hold our bookmarklet UI and using django templates (the {%include … %} part) we can template the html easily, inheriting the context of this view. Please make sure to add the “spaceless” tags here because the innerHTML will fail if there are line breaks and I’ll be damned if I have to add linebreaks to my template. Also be extremely picky with your quotes as that can also lead to disaster.

Now in our “bookmarklet_body.html” we can create our UI just as if it were any other django template (almost). For example:

{% load static %}
{% load url_tags %}
{% comment %}NOTE Make sure you ONLY use double quotes!{% endcomment %}    

<div id="bookmarkletAssistant">
    <img src="{% server_scheme_and_netloc %}{% get_static_prefix %}images/assistant.png" style="border: 2px solid #666666;"/><br/>    
    <img id="bookmarkletLoadingImage" src="{% server_scheme_and_netloc %}{% get_static_prefix %}images/bookmarklet-ajax-loader.gif" style="display:none;"/>
</div>
<div id="bookmarkletResponse">
    <p>Hey Chief, almost ready!</p>
    <div id="bookmarkletPictureSelector">
        <p>
            <label>Select Picture:</label><br/>
            <div id="bookmarkletImageContainer">
                <img id="bookmarkletCurrentImage" />
                <a id="bookmarkletSelectPrev">&laquo;</a><a id="bookmarkletSelectNext" >&raquo;</a>
            </div>
        </p>
    </div>
    <p id="bookmarkletImageCheckboxContainer"><input id="bookmarkletImageCheckbox" type="checkbox" style="margin-right: 5px;" />
    <label>No Picture</label></p>
    <p>
        <label>Editor&rsquo;s Note:</label><br/>
        <textarea id="bookmarkletTextArea" placeholder="Add your commentary on the article here. (optional)"></textarea>
    </p>
    <input id="bookmarkletSubmit" type="submit" value="Publish Article" />
    <a id="bookmarkletCancel">Cancel</a>
</div>

Notice how all the elements are contained within elements with IDs. This is extremely important so you can clean up after yourself and not pollute the dom. You will also need to do this so that subsequent interactions can find your bookmarklet and update the contents. BTW this is exactly what is used in Edorati. 

Make sure that you have a view set up so that you know it is from your bookmarklet and can respond with your javascript template. When the user initializes the bookmarklet (see part 1 for reference) they will be requesting a url that you control. All you need to do is map that url to a view function and do the following:

    variables = {'some_var': 'blah blah'}
    resp = render_to_string('bookmarklet.js', variables)
    return HttpResponse(resp, mimetype="text/javascript")

This will make sure we are responding with javascript and not html. The javascript will be executed as soon as it is received by the user. It’s actually very fast and should feel close to instantaneous. 

Making it Look Pretty

Plain unstyled html would be a terrible idea for a bookmarklet. Keep in mind that you are injecting onto another page where their stylesheet is unknown. That means we need to style and reset the css for the bookmarklet or it will not look consistent across sites. Here’s how:

    // in bookmarklet.js
    t=d.createElement("link"), 
    t.rel="stylesheet";
    t.media="screen, projections";
    t.type="text/css";
    t.href="{% server_scheme_and_netloc %}{% get_static_prefix %}stylesheets/bookmarklet.css";

Whoa, dynamically loading css just for our bookmarklet! Nothing fancy going on here, just using plain ol’ css to beautify our bookmarklet. Just make sure that you do not inadvertently style the rest of the elements on the page with your stylesheet. As mentioned before, add an ID for all major elements and when it comes to css, use the ID selector to style the element instead of the element selector or class. If you have to use classes just obfuscate the name so there is little possibility of you impacting the page i.e bookmarklet-textarea-field vs. textarea.  

Next Week

Next week we’ll go over how to make the bookmarklet come alive with interaction through event binding and requests using about 15 lines of pure javascript and no libraries.

Adding a UUID Field to an Existing Django Model

It’s not immediately obvious how to add a UUID field to an existing Django model. Let’s say you have a database with some data in it about certain accounts, but you realize you need a way of obfuscating the pk of a model instance. UUID is a perfect candidate and a pretty popular implementation for Django comes from dcramer called django-uuidfield. Everything is fine and dandy until you try to add that field to an existing model that already has data, then try to auto migrate it using south. 

How to add a UUID field to an existing model

The problem here is that we need to add a UUID value to all existing records, not just new ones. Peaking through the source code you will see what uuid method they are using and what the data should look like.

  1. Add the UUID field to your model with blank=True, null=True, max_length=32, auto=True in models.py.
  2. Run the schemamigration command with south and then open up the resulting migration file. 
  3. Edit your migration file with the following:
# You'll need to import the following
import uuid

# At the end of the forwards() function add the following
def forwards(self, orm):
    ...
    for a in MyModel.objects.all():
        a.uuid = u'' + str(uuid.uuid1().hex)
        a.save()

That will loop through all the existing instances of that model in your database and add a uuid to it as part of the migration. Make sure you test it first!

How to make an interactive bookmarklet part 1

Here’s how to put together a non-trivial interactive bookmarklet. For some reason there are not too many resources on the subject, but it has become a much more popular way of user’s interacting with a service without having to be on the site. Pinterest, Evernote, etc. all rely heavily on the ease of use of bookmarklets. 

For Edorati, I needed to create a simple, fast way for user’s to publish an article to their online newspaper whenever they came across something they liked. It also had to look nice and allow them to select a picture thumbnail and add an editor’s note. Oh yeah it also needs to be upgradeable from the server side so that the user doesn’t have to reinstall it every time you want to add something new. 

Why is it so hard?

Bookmarklets are especially hard for a couple reasons:

  1. You don’t own the page the bookmarklet will deploy on
  2. Cross-domain requests
  3. Maintaining some level of security
  4. Everything needs to be manipulated through javascript
  5. Avoid using full js libraries like jquery

What’s the plan?

I ended up building a simple framework for bookmarklets and that uses django to process the request, handle errors, and provide responses. The bookmarklet is actually just a small piece of javascript code that gets executed as soon as it is clicked. It will tell my server that it’s ready to be initialized for a particular user. Then the server will make a response with ONLY javascript that is contained in a self executing function which will manipulate the target page the user is on. From there any interactions are a matter of building a request on the client side or returning a javascript response from the server-side. Sound simple? Not really…

How do I make cross-browser requests, do I use Jquery?

No Jquery! We’re going to avoid all libraries and roll our own. Adding big libraries will make your bookmarklet slow and you probably only need a couple of things from it. All we need in this case is a small workaround to make cross-browser requests right from the user’s page. This is the pattern:

javascript:(function(){
    var d=document,
    // Create a script element
    s=d.createElement("scr"+"ipt"),
    // Set the src to an endpoint on your server
    s.src = "http://www.urltoyourapi.com/?param1=something&param2=somethingelse";
})()

Now your server can respond with javascript to fill in that script element that was just created on the user’s page. By adding some GET parameters to the end of the URL you also have a way of communicating information about the user or the page. For example you may want to construct the url in a way that it passes the users obfuscated ID number so you can save information to that user’s account. This is all a bookmarklet is. 

Making it future proof

To prevent your users from having to install the bookmarklet every time you want to make a change, you should not include any presentation or interactive logic in the bookmarklet. For Edorati, all I need to know from the bookmarklet is who is the user. All the bookmarklet needs to do is essentially tell the server that it wants to be initialized. Everything else should happen on the server side. That way you can make any changes you want to the look or the functionality without ever having to change the user’s bookmarklet that they installed. 

Next post will discuss constructing templated javascript on the server side to make a bookmarklet interactive, clean, and safe…

Add fields to a Django serializer

Here’s a quick and dirty way to add fields to a serializer. I wanted to add in the absolute url to the json data that was being sent back during an ajax request. I wouldn’t recommend it for large querysets since you are essentially adding a json load and another dump, but for a single model instance I think it’s ok. The alternative would be to use a library like wadofstuff-django-serializer, but I really want to avoid a dependency for just a single purpose.

import simplejson as json
from django.core import serialize

# Somewhere in your view
# Note that I'm only serializing a single instance hence the tup
resp = serializers.serialize("json", (article,))

# Add absolute url to
resp_obj = json.loads(resp)
resp_obj[0]['fields']['get_absolute_url'] = article.get_absolute_url()
r = json.dumps(resp_obj)
# Then respond back as usual with your updated json object

Cohort Analysis in Django

I wrote a simple reusable app for doing cohort analysis in Django. Cohort analysis is an incredible tool for deflating your ego. Since it compares apples to apples you will see if you are actually helping, hurting or, more than likely, making no impact at all on your users with the iterations you’ve been making on you’re awesome web app. 

What is Cohort Analysis?

Cohort analysis involves segmenting your users into smaller groups so you can compare measurements on them. For example, a cohort could be the user’s that signed up for your app in a week. So all the people that joined last week is a group, everyone who joined two weeks ago are in a group, etc. Now we can analyze actions that the cohort has taken in relation to the week they joined. For example “Users who joined the week of 2/1 performed 10% more action x in their first week than the users who joined in 1/25.” You can then think about the changes you made and correlate it back to the results. This prevents things like traffic spikes or that PR you did from messing up your numbers. You’ll be able to see over time if you are actually getting any better rather than looking at aggregate statistics of the activity of all your users. If you try hard enough with any aggregate stat you can make it look like a hockey stick. But that’s bullshit because you have no idea if the changes you made are the cause of that improvement when you look in the aggregate. 

Cohort Analysis in Django

So how do we do this in django? I wrote a very simple, reausable app called django-cohorts. The objective is to set up a simple framework for running cohort analysis defined by the segments (cohorts) you want and the metrics you want in a dashboard that lives in the admin section. It’s raw right now, but it’s a start. 

How does it work?

I’ve done the base cohorts by week, which is pretty common, but you can override that. In metrics.py, create a class with functions that will return a percentage. It will send you the users and you do some db lookup for certain actions the user took. After you’ve hooked it up to your url conf, you can then see the analysis with a dropdown of the metrics that are available.

Looking for Contributors

If anyone wants to help make this more sophisticated please let me know. Feel free to fork it on GitHub.

A Note about Cohort Analysis

Since cohort analysis typically looks at usage over time, you need to have some data for it to be useful. You can’t identify patterns that are significant without at least a few weeks of data. Just saying.

How to overwrite the save of a Django form like a pro

Use the “super” function to use the save method built into the form plus the extra things you need. Example:

class DebateMediaForm(forms.ModelForm):
    def __init__(self, *args, **kwargs):
        super(DebateMediaForm, self).__init__(*args, **kwargs)
        self.fields['image'].required = True
        self.fields['source'].label = "Image Source"
        
    class Meta:
        model = MediaContent
        fields = (
            'image',
            'source',
        )
    def save(self, user, debate):
        media = super(DebateMediaForm, self).save(commit=False)
        media.added_by = user
        media.related_debate = debate
        media.content_type = "P"
        media.save()
        return debate

This is really handy for dealing with a ModelForm where you only want to have the user fill out a couple fields, but additional background information is required. In this case I need to specify who added the media. That’s something you don’t want your users to have to fill out, but you still want the luxury of using a model form because of the beautiful built in save function. Just super that shit and move on :)