Backbone in the wild, lessons learned using backbone.js

This week I finished up a major redesign of Edorati which helps people discover articles with their friends. The reason I ended up going with a client-side MVC framework, Backbonejs, is because a huge piece of the existing web app was long javascript files with tons of jquery event handling. In short, it was a mess and difficult to build on top of. Although I didn’t set out to use backbone for everything, I ended up doing just that because it made modular development of client side code that much better. It made it feel like javascript was as real as the python/django that powers it. 

Getting Your Head Around Client-Side MVC

Coming from a world of server-side templating, models, and views with Django, I had to start thinking of how that translates to backbone. Your backend will mostly be an API that serves data based on the requested URL and your business logic. Using backbone, my django project only serves one template for the whole app. Everything else is handled by backbone. That should give you an idea of how much heavy lifting backbone is able to do. I ended up stripping 70% of the view code I had written in django. 

Oversimplified Explanation of Backbone

Here is my grossly oversimplified mental model of what backbone is. 

Backbone organizes your front-end javascript code into an MVC style framework. There are 5 primary components to backbone:

  • Models: serialized data that describes an instance (similar to a django model without the database instructions)
  • Collections: a list of models (kind of like a queryset)
  • Views: a handler for a section of your app (similar to a django view, but typically more granular)
  • Router: routes a url to the appropriate view using a hashbang or pushState
  • Templates: Sticks you data into html. Not a part of backbone, but you will almost certainly use them. I personally use handlebars, but you can use whatever you want.

Backbone Patterns No One Tells You About

Going through the tutorials and reading other people’s code I’ve picked up on some basic patterns that are used by convention. Some of them were not immediately obvious when I started learning. Others I learned the hard way.

  1. Granular Views - Consider a page that has a list of articles. With django you might create a view that grabs a queryset and outputs a template with a for loop to produce the list. You might then use jquery to handle events like the user editing that instance. With backbone you would set it up like this: A list view creates a collection of articles, for each model in the collection create an article view. Notice that there are two new pieces in there. The collection which is tied to a url which will call your api then automatically create a list of models from the received data and a view for an individual article. It was not obvious to me why in the world you would want a view for an individual article which might be 5 lines of code. The reason is for event handling. You’re basically setting the scope of events that apply to that code. This will help you avoid large, complicated views that try to handle all sorts of things for individual items. In the long run this leads to better separation of the different parts of your app and makes it much more manageable. Seems like more work up front, but it makes it easier in the end. 
  2. Zombies - I wish the documentation was more up front about this. Any non-trivial app typically creates and changes many views within a single session. YOU HAVE TO CLEAN THAT SHIT UP. If you don’t, your beautiful modular code will exhibit bizarre behavior as soon as you create a new view without first removing the old one. Events are delegated to elements. Even though you may have cleared the html, you have not removed the delegated events. Before you instantiate a new view that replaces another one, remove it and unbind all events from it. (this.remove() this.unbind()). It’s really hard to debug this, but the symptoms are usually things like the wrong ID being used, wrong model is changing etc. Zombies. For more info read this.
  3. Requirejs - Non trivial apps will need to be separated into multiple files. The tutorials out there fail in explaining this. You will almost certainly be needing requirejs. Learn require well. Took me an embarrassing amount of time for me to get this.
  4. One view per file - Related to the requirejs tip above, you only put 1 view in per file. Coming from django I just thought I would put a whole bunch of views in one file, but that’s not how it’s done with backbone using require. One view, model, collection, per file. Why? Because using require you need to return a class and it can’t handle you returning multiple classes. 
  5. Initializing an app - Using require, it’s not obvious how you “boot” the whole app. Create a view that will represent your app and create an initialize function to do what you need to start it up. In your main.js (used by require) create an instance of your app and voila, it will initialize. Typically you would initialize the router and any views that must be there no matter what. For Edorati that means the sidebar and router.
  6. Router is your friend - Let your router handle the switching of views. Don’t try to stick the router logic into your views. Need to send people to a profile detail page? Create a route for it and use that url in the template. Don’t create an event handler in your view so that when someone clicks on it it creates a new view. Just use the router for what it does, route things. Doing this also means you’ll have re-usable links that can be used outside of the session. i.e. link that directly takes you to a user’s profile.

I’m sure I’ll think of more, but that’s it for now. Most of that list will not make sense until you dive into backbone creating your app. Tweet me if you have any questions @alexkehayias.

Write a script for non-trivial things you do more than once

I just wasted four hours migrating data semi-manually to a new server for a huge update to Edorati only to find that I made a mistake and all my work was rendered useless. What’s the problem here? The server? The data? No. Shit happens. Mistakes happen.

The moral of the story is if you have to do something more than once that involves a non trivial procedure just write a damn script. 

Unfortunately, we have a bias against doing any additional work if we perceive to only need to do something a few times in an ideal scenario. The problem is that reality and ideal scenarios don’t usually co-exist. I should have been able to brush it off, mutter some choice swear words, and just re-run my script on a new server, but instead, I’ve been sitting hear bitching and complaining that the world as I know it is a soul-less, horrible, Bieber-filled place. 

I ended up having to write the script anyway. It took me 20 minutes. It will still take a good 30 minutes for the script to run, but my involvement would go from 4 hours to about 5 minutes. 

I’m going to go lay down on some train tracks for being so dumb.

My AngelHack Project, AlertBox, alerts for your social life

Had a great time with everyone this weekend at AngelHack, a startup hackathon with teams from all over the country. I worked on a little app called AlertBox to notify people of important updates from their social networks as soon as it happens. What if you could get alerted when your friend who just moved half way around the world is having a bad day? What would you do with that timely information?

What if we got the right information from our social networks just in time? I think the world could be a better place for it. It’s just a weekend hack, but the idea is powerful. What do you think?

Oh and it’s a python/django/backbone app. Enjoying backbone recently, will make an appearance in my portfolio much more often methinks.

6% Better than Twitter

The past two weeks I’ve been doing lots of pitching and am reminded just how important the question “what makes you unique?” is so important. If you’re answer is not a narrative about how you’re 10 times better than anything else the world has to solve a problem you will lose. 

Incremental Improvements vs 10x Game Changers

When I was in DC last week I had a chance to see Dan Yates from opower present. He talked about the power of incremental improvements and how in the macro world they can represent a huge opportunity. For opower that means reducing power consumption for it’s clients (power companies) by 6%. Very different than a clean tech startup that is trying to replace power plants with solar/wind/whatever. That small change, in aggregate, is a huge improvement and they are able to capture some of the value they are creating (profit).

Then there are consumer web startups where an incremental improvement of 6% over anything is laughable. Imagine someone pitched their startup to you as “I’ve built a product that’s 6% better than Twitter.” You just can’t get excited about an incremental improvement when the web is this crowded. In one of Peter Thiel’s class note essays (go read all of them), Peter talks about dominating markets and becoming the last mover. Competition leads to incremental improvements. Domination comes from 10x game changers.

Don’t Compete, Dominate

There are plenty examples of incremental improvements leading to big impacts (Edison’s light bulb is an incremental improvement over what existed), but in the crowded world of web startups I feel that incremental won’t cut it. If you want to dominate you need to be a 10x improvement. Obviously if you’re not solving the right problem you can be 10x improvement on something no one cares about, so keep that in mind too. The point is to dominate not compete. 

Lean Startups and Domination

I’m beginning to see how lean startup tactics are indoctrinating us towards incremental improvements. Maybe you have a big idea and vision of how you will change the world. You dissolve that into a testable series of assumptions and set off to prove it. As you go along and test, speak with people, and iterate you increment. You hone in on the actual problem and optimize different parts of your funnel. However, it’s too easy to fall in love with the tests, data, and incremental improvements. Sometimes the best edit is a rewrite (I think PG said that). Sometimes your pivot is just an incremental improvement. We need to think less about optimization and more about domination. You improved your signup conversion rate by 5%? Cool, but you still only have 100 users because no one cares, you’re not can’t miss, not novel, not 10x, just noise.

Where do we go from here

Maybe it’s time we start thinking about intelligent design rather than evolution (not in the education/political sense). If we start thinking about domination from day 1, how does that change our approach to product and distribution? There’s the danger of solving the wrong problem, but there’s also the problem of not solving the problem hard enough. I already know how to optimize, but I don’t know how to dominate. I’m going to work on that.

Advice for submitting Open Graph apps to Facebook

I recently had to submit an app with two Open Graph actions to Facebook for their blessing. It’s a clunky process where you need to have it fully working on externally available server (in my case a dev server), post the action to your account successfully, then tell them how to test it so that it meets their guidelines. Turn around time was about 2 days. 

And then they rejected an action

I recently did a deeper integration with Facebook for Edorati to bring some social reader functionality so you can follow what your friends are reading more easily. One of the actions was approved right away, but the “read” action was rejected. OMG wut? It’s so far within their guidelines I could puke and it would still be kosher.

After thinking it through, I decided that my integration was flawless (I’m not actually as cocky as my writing presumes) and within guidelines. I resubmitted the same thing, no code changes, nothing. All I did was update my instructions for them to test it to specifically hit every point in their guidelines and it was approved.

Lesson

Avoid wasting time with the approval process by being very specific about your test case when submitting your app. Clearly hit on points such as global settings to turn the action on or off, per item control of the action (in my case, that’s a button for sharing an article on the article detail page), why it’s relevant to other people, and make your test case super easy to follow. Oh, it also helps if your code actually works ;-)

Here’s the Edorati submission (sanitized):

Log in to DEV_URL with username: USERNAME, password: PASSWORD. Go to ARTICLE_PAGE and click the “mark as read” button. This will perform the “read” Facebook open graph action. Clicking the button again will delete the post. Go to SETTINGS_PAGE to toggle the “read” action globally on or off. The read action is performed when a user reads an article which is a strong indicator that this is a relevant article that other people will want to read in their network. There is global control for this action as well as per article with the Mark as Read/Unread button on the article detail page. 

How to make an interactive bookmarklet part 3

In the last two parts of this series we learned about the framework for building bookmarklets and how to use javascript with django templates for optimal maintainability. Now let’s look at how to make interactions work without needing any external javascript libraries. 

Why not include Jquery

My guess is that all you really want are event bindings and for that you don’t need an entire library. Granted, lots of user’s have jquery cached in their browser already, but if not you have to include that which will make your bookmarklet slower. Every library you add will add more “weight” to the overall bookmarklet. Read the next paragraph before you decide you need the whole jquery library or you can make do with 6 lines of pure javascript.

Binding Events to Handlers

So you want to be fancy and make the bookmarklet change when the user clicks on something? Tough shit. Just kidding. The problem with event handling is that making it work cross-browser is tough. Jquery helps make writing js more cross-browser compatible and that’s a huge part of it’s overall value.

Here’s how we can add event handlers that will work across all browsers without Jquery. I can’t find where I originally found this from, but when I do I’ll add an attribution link.

    var bindEvent = function(elem, evt, cb) {
        //see if the addEventListener function exists on the element
        if ( elem.addEventListener ) {
            elem.addEventListener(evt,cb,false);
        //if addEventListener is not present, see if this is an IE browser
        } else if ( elem.attachEvent ) {
            //prefix the event type with "on"
            elem.attachEvent('on' + evt, function(){
                //use call to simulate addEventListener
                cb.call(event.srcElement,event);
            });
        }
    };

Now we have a simple way to make interactions happen within the bookmarklet itself. For example:

    // Remove the bookmarklet from the document assuming b is body and c is the bookmarklet
    var cancelDialog = function(e){
        b.removeChild(c);
    };
    bindEvent(cancelButton,"click", function(e){
        cancelDialog(this);
    });

All that without including the whole jquery library! That will keep your bookmarklet lightweight and fast. 

Submitting a Form

Using our event binding snippet above we can create a simple handler for submitting a form free from cross-domain errors. Those of you familiar with jsonp will see that this is very similar, but done manually and in a simpler way.

    // Assuming we have the following in the script
    var d=document,
    s=d.createElement("scr"+"ipt");
    b.appendChild(s);
    
    // Bind a submit event to a button click
    var submitArticle = function(e){
        // get the values of any form data you need
        // URI encode the variables as GET parameters
        s.src = "http://www.urltoyourservice.com?var1=" + encodeURIComponent(variable1) + "&var2=" + encodeURIComponent(variable2);
    };
    
    bindEvent(submitButton,"click", function(e){
        submitArticle(this);
    });

This will make a GET request to your your server with the parameters of your form. We can now process that request and return a response just like we did to initialize the bookmarklet in the first place. All we need to do is respond back with javascript. You can do this numerous times. All your doing is manipulating the bookmarklet that is already a part of the document. Make sure that you clean up after yourself (see my earlier example of binding a cancel button to removing the bookmarklet from the document). 

There you have it. An interactive bookmarklet that’s fast, maintainable, and lightweight. You can make changes to it at any time since the initialization happens server-side with familiar templates. You can make cross-domain requests without any errors. You can even bind events and handle them all without needing Jquery. 

Enjoy!

How to make an interactive bookmarklet part 2

Let’s talk about how to generate a bookmarklet using server-side templating and pure javascript (no libraries!). In part 1 we discussed the basic framework for making a bookmarklet and performing requests directly from the client’s page. Now we’ll see how to set up the foundation for a bookmarklet UI and make it look good.

Building a Bookmarklet Template in Django

Let’s create a file called bookmarklet.js. In it we are going to add a self executing piece of javascript and leverage django templates to fill in the information we need. In here we are going to build the response using nothing but javascript and send it back to the user who requested the bookmarklet (remember all they did was request it to be initialized, see part 1 for more info). With django templates we can do some pretty handy things so that you don’t need to manually write out the template as a string in javascript. For example:

javascript:(function(){
    var d=document,
    {% comment %}Create the elements we need {% endcomment %}
    c=d.createElement("div"),
    b=d.body,
    l=d.location;
    {% comment %} server_scheme_and_netlog is my own tag which returns the base url of the website{% endcomment %}
    c.id="bookmarkletArticleAlertBox";
    {% comment %}This is where django templates get's awesome{% endcomment %}    
    {% comment %}Make sure the html rendered has no breaks{% endcomment %}    
    c.innerHTML = '{% spaceless %}{% include "bookmarklet_body.html" %}{% endspaceless %}';
    {% comment %}Add the elements to the dom{% endcomment %}
    b.appendChild(t);
    b.appendChild(c);

....snip....

What we did was construct a div that will hold our bookmarklet UI and using django templates (the {%include … %} part) we can template the html easily, inheriting the context of this view. Please make sure to add the “spaceless” tags here because the innerHTML will fail if there are line breaks and I’ll be damned if I have to add linebreaks to my template. Also be extremely picky with your quotes as that can also lead to disaster.

Now in our “bookmarklet_body.html” we can create our UI just as if it were any other django template (almost). For example:

{% load static %}
{% load url_tags %}
{% comment %}NOTE Make sure you ONLY use double quotes!{% endcomment %}    

<div id="bookmarkletAssistant">
    <img src="{% server_scheme_and_netloc %}{% get_static_prefix %}images/assistant.png" style="border: 2px solid #666666;"/><br/>    
    <img id="bookmarkletLoadingImage" src="{% server_scheme_and_netloc %}{% get_static_prefix %}images/bookmarklet-ajax-loader.gif" style="display:none;"/>
</div>
<div id="bookmarkletResponse">
    <p>Hey Chief, almost ready!</p>
    <div id="bookmarkletPictureSelector">
        <p>
            <label>Select Picture:</label><br/>
            <div id="bookmarkletImageContainer">
                <img id="bookmarkletCurrentImage" />
                <a id="bookmarkletSelectPrev">&laquo;</a><a id="bookmarkletSelectNext" >&raquo;</a>
            </div>
        </p>
    </div>
    <p id="bookmarkletImageCheckboxContainer"><input id="bookmarkletImageCheckbox" type="checkbox" style="margin-right: 5px;" />
    <label>No Picture</label></p>
    <p>
        <label>Editor&rsquo;s Note:</label><br/>
        <textarea id="bookmarkletTextArea" placeholder="Add your commentary on the article here. (optional)"></textarea>
    </p>
    <input id="bookmarkletSubmit" type="submit" value="Publish Article" />
    <a id="bookmarkletCancel">Cancel</a>
</div>

Notice how all the elements are contained within elements with IDs. This is extremely important so you can clean up after yourself and not pollute the dom. You will also need to do this so that subsequent interactions can find your bookmarklet and update the contents. BTW this is exactly what is used in Edorati. 

Make sure that you have a view set up so that you know it is from your bookmarklet and can respond with your javascript template. When the user initializes the bookmarklet (see part 1 for reference) they will be requesting a url that you control. All you need to do is map that url to a view function and do the following:

    variables = {'some_var': 'blah blah'}
    resp = render_to_string('bookmarklet.js', variables)
    return HttpResponse(resp, mimetype="text/javascript")

This will make sure we are responding with javascript and not html. The javascript will be executed as soon as it is received by the user. It’s actually very fast and should feel close to instantaneous. 

Making it Look Pretty

Plain unstyled html would be a terrible idea for a bookmarklet. Keep in mind that you are injecting onto another page where their stylesheet is unknown. That means we need to style and reset the css for the bookmarklet or it will not look consistent across sites. Here’s how:

    // in bookmarklet.js
    t=d.createElement("link"), 
    t.rel="stylesheet";
    t.media="screen, projections";
    t.type="text/css";
    t.href="{% server_scheme_and_netloc %}{% get_static_prefix %}stylesheets/bookmarklet.css";

Whoa, dynamically loading css just for our bookmarklet! Nothing fancy going on here, just using plain ol’ css to beautify our bookmarklet. Just make sure that you do not inadvertently style the rest of the elements on the page with your stylesheet. As mentioned before, add an ID for all major elements and when it comes to css, use the ID selector to style the element instead of the element selector or class. If you have to use classes just obfuscate the name so there is little possibility of you impacting the page i.e bookmarklet-textarea-field vs. textarea.  

Next Week

Next week we’ll go over how to make the bookmarklet come alive with interaction through event binding and requests using about 15 lines of pure javascript and no libraries.

Adding a UUID Field to an Existing Django Model

It’s not immediately obvious how to add a UUID field to an existing Django model. Let’s say you have a database with some data in it about certain accounts, but you realize you need a way of obfuscating the pk of a model instance. UUID is a perfect candidate and a pretty popular implementation for Django comes from dcramer called django-uuidfield. Everything is fine and dandy until you try to add that field to an existing model that already has data, then try to auto migrate it using south. 

How to add a UUID field to an existing model

The problem here is that we need to add a UUID value to all existing records, not just new ones. Peaking through the source code you will see what uuid method they are using and what the data should look like.

  1. Add the UUID field to your model with blank=True, null=True, max_length=32, auto=True in models.py.
  2. Run the schemamigration command with south and then open up the resulting migration file. 
  3. Edit your migration file with the following:
# You'll need to import the following
import uuid

# At the end of the forwards() function add the following
def forwards(self, orm):
    ...
    for a in MyModel.objects.all():
        a.uuid = u'' + str(uuid.uuid1().hex)
        a.save()

That will loop through all the existing instances of that model in your database and add a uuid to it as part of the migration. Make sure you test it first!

How to make an interactive bookmarklet part 1

Here’s how to put together a non-trivial interactive bookmarklet. For some reason there are not too many resources on the subject, but it has become a much more popular way of user’s interacting with a service without having to be on the site. Pinterest, Evernote, etc. all rely heavily on the ease of use of bookmarklets. 

For Edorati, I needed to create a simple, fast way for user’s to publish an article to their online newspaper whenever they came across something they liked. It also had to look nice and allow them to select a picture thumbnail and add an editor’s note. Oh yeah it also needs to be upgradeable from the server side so that the user doesn’t have to reinstall it every time you want to add something new. 

Why is it so hard?

Bookmarklets are especially hard for a couple reasons:

  1. You don’t own the page the bookmarklet will deploy on
  2. Cross-domain requests
  3. Maintaining some level of security
  4. Everything needs to be manipulated through javascript
  5. Avoid using full js libraries like jquery

What’s the plan?

I ended up building a simple framework for bookmarklets and that uses django to process the request, handle errors, and provide responses. The bookmarklet is actually just a small piece of javascript code that gets executed as soon as it is clicked. It will tell my server that it’s ready to be initialized for a particular user. Then the server will make a response with ONLY javascript that is contained in a self executing function which will manipulate the target page the user is on. From there any interactions are a matter of building a request on the client side or returning a javascript response from the server-side. Sound simple? Not really…

How do I make cross-browser requests, do I use Jquery?

No Jquery! We’re going to avoid all libraries and roll our own. Adding big libraries will make your bookmarklet slow and you probably only need a couple of things from it. All we need in this case is a small workaround to make cross-browser requests right from the user’s page. This is the pattern:

javascript:(function(){
    var d=document,
    // Create a script element
    s=d.createElement("scr"+"ipt"),
    // Set the src to an endpoint on your server
    s.src = "http://www.urltoyourapi.com/?param1=something&param2=somethingelse";
})()

Now your server can respond with javascript to fill in that script element that was just created on the user’s page. By adding some GET parameters to the end of the URL you also have a way of communicating information about the user or the page. For example you may want to construct the url in a way that it passes the users obfuscated ID number so you can save information to that user’s account. This is all a bookmarklet is. 

Making it future proof

To prevent your users from having to install the bookmarklet every time you want to make a change, you should not include any presentation or interactive logic in the bookmarklet. For Edorati, all I need to know from the bookmarklet is who is the user. All the bookmarklet needs to do is essentially tell the server that it wants to be initialized. Everything else should happen on the server side. That way you can make any changes you want to the look or the functionality without ever having to change the user’s bookmarklet that they installed. 

Next post will discuss constructing templated javascript on the server side to make a bookmarklet interactive, clean, and safe…

Add fields to a Django serializer

Here’s a quick and dirty way to add fields to a serializer. I wanted to add in the absolute url to the json data that was being sent back during an ajax request. I wouldn’t recommend it for large querysets since you are essentially adding a json load and another dump, but for a single model instance I think it’s ok. The alternative would be to use a library like wadofstuff-django-serializer, but I really want to avoid a dependency for just a single purpose.

import simplejson as json
from django.core import serialize

# Somewhere in your view
# Note that I'm only serializing a single instance hence the tup
resp = serializers.serialize("json", (article,))

# Add absolute url to
resp_obj = json.loads(resp)
resp_obj[0]['fields']['get_absolute_url'] = article.get_absolute_url()
r = json.dumps(resp_obj)
# Then respond back as usual with your updated json object