Alex Kehayias's Blog

Importing Postgres Data Into Hadoop HDFS

Wed, 27 Feb 2013 13:54:30 -0500

Recently I’ve been getting a crash course in big data. I ran into an issue where the analytics that used a certain table on out Postgres database was so large it would not run. Like to the point where it would grind a 8gb of memory server to a near halt on a single query. While I had gone through several optimizations, it still was not performant enough at the scale needed. In fact it caused havoc with our host due to IO issues. We’re talking 50 million rows or more that need to be analyzed in a scalable way.

Enter Hadoop

I’m sure Hadoop needs no introduction for most, but the great benefits is a scalable way to take massive amounts of data and answer the questions you have.

How do you get existing data into Hadoop?

I had a ton of data that I needed to get into HDFS that was on a Postgres server. I needed a way to dump the data into a useable format so that I could use it properly with Hadoop. Luckily there is an Apache project called Sqoop (yes Sqoop Hadoop is just an amazing combination of words).

Running Sqoop Jobs

Sqoop requires that you have Java and Hadoop installed on the machine that is going to run the job. The great news is that it doesn’t have to be on the same machine as your server. You can run it from any machine, but you typically will run it from a Hadoop cluster. That means it doesn’t matter where your server lives or who if your host doesn’t allow you to install your own packages. You also don’t have to dump the data to the machine that the Sqoop job will be running on. In my case I wanted all my data in S3 as a super portable replacement for HDFS.

Unfotunately, it took a lot of trial and error to get it working. Make sure that you install the proper JDBC driver by putting it on your java classpath or put it in the sqoop directory (usually in /usr/bin/sqoop).

Here’s a local example using Postgres:

    sqoop import --connect jdbc:postgresql://localhost:5432/mydb \
                      --table my_table \ 
                      --username myusername \
                      --password mypassword \
                      --direct

This will dump the table my_table on a local postgres server. The —direct part is a speedup that’s unique to Postgres and MySql connectors for sqoop.

To dump data directly to S3:

sqoop import --username myusername --password mypassword --table my_table --target-dir s3n://MYS3APIKEY:MYS3SECRETKEY@bucketname/folder/ --hive-drop-import-delims --escaped-by "\\"

Some notes to help your blood pressure:

Make sure your bucket doesn’t have an underscore in it. For some reason I wan’t able to get it to work if there was an underscore in the bucket.
Make sure you use trailing slashes. This should help your blood pressure.
—direct doesn’t work with —escaped-by
If you have freeform text fields in your data and there is \n it will fuck up your Hadoop job if you try to read new lines of data by splitting on \n. Use the —hive-drop-import-delims to parse those out. I don’t know why it works, this has nothing to do with Hive, it just cleans up your output.
—escaped-by should always be used if you have freeform text in your data. This will help you parse the data when you run a job because you can ignore important characters (such as a comma) if they are escaped, letting you split data into fields more easily.

Appending Data

What happens if we need to get updated data off of our database? HDFS doesn’t really do an “update” to existing data, but we can append all the new rows added past a certain point. To do that we can use the —incremental argument.

sqoop import --connect jdbc:postgresql://localhost:5432/mydb --username myusername --password mypassword --table my_table --incremental append --check-column id  --hive-drop-import-delims --escaped-by "\\"

Append Data To S3 Using Sqoop

I wanted to keep my data independent of the cluster that would run the job. This gives you flexibility to run your jobs wherever, on a cluster or on something like Elastic MapReduce. You have the freedom to build and destroy clusters on a whim because there is no persistant data to lose.

There’s no point in appending if you don’t know the last thing you appended (which row was last to go in the database). Sqoop has a “job” command that will store the last id (or date) of the last items dumped. It then queries only the rows after that point. Very cool!

Unfortunately (again), I couldn’t get this to work by dumping directly to S3 using the —incremental argument so I had to use a workaround. First run the command then use distcp to copy the data to S3. For some reason, incremental appends using S3 as the —warehouse-dir does not work even when specifying a -fs argument.

Hopefully this saves you as much time as I spent figuring it out.

First save the job:

sqoop job --create myjob -- import --connect jdbc:postgresql://myaddress:12345/dbname --table mytable --hive-drop-import-delims --escaped-by "\\" --username myusername --password mypassword --incremental append --check-column id --split-by id

Then run it:

sqoop job --exec myjob

Then copy it in a parallelized way to S3:

hadoop distcp mytable s3n://MY_S3_KEY:MY_S3_SECRET@mybucket/folder/

Boom, data on S3 with no dependency on your cluster living another day! I should note that you will want to use a centralized metastore for saving the last rows run by append. See the sqoop docs for more info.

Django model creation shortcut with kwargs

Fri, 16 Nov 2012 10:26:00 -0500

Let’s say you’ve got this model in Django that you need to create that has a lot of fields. You’ve already mapped out the data, but you think you need to write in each argument in the Model.objects.create method. Thankfully, these methods allow **kwargs to be passed in so you can do this:

# Remember that kwargs is just a dictionary that's mapping
# arguments to values (in this case model fields to values)
MyModel.objects.create(**some_dict)

# Or if we are updating it
MyModel.objects.update(**some_dict)

# Conveniently we could do something like this,
# assuming your field names are mapped exactly to the model names.
# Sometimes you don't want a model form so this works.
MyModel.objects.create(**request.POST)

# Oh, but I have a foreign key! No prob. 
MyModel.objects.create(
    fk_field = some_object,
    **some_dict
)

Django Gotcha: Duplicate models when using get_or_create

Fri, 19 Oct 2012 08:24:00 -0400

When you’re saving some object to the db, there’s a gotcha with DateTimeFields that drove me nuts. It took me way too long to figure this out, but if you have a DateTimeField with the option auto_now_add, it will supersede a datetime object in a Model.objects.get_or_create call. This means that if you are tying to prevent duplicates based on date (say you’re saving some data from an API that has a timestamp) you will still get duplicates. It’s in the docs, but not the implications when using get_or_create. Check out this example:

# Given this simple model
class Foo(models.Model):
    name = models.CharField(max_length=100)
    date_added = models.DateTimeField(auto_now_add=True)

# This will always be true, even if an instance
# with this name and today's date already exists

bar, created = Foo.objects.get_or_create(
    name = 'Alex', 
    date_added = some_datetime_obj
)

print created
# >> True


# The problem is, auto_now_add does some stuff that 
# makes it uneditable, and fucks up my expectations
# when using it with get_or_create

# Here's the solution
class Foo(models.Model):
    name = models.CharField(max_length=100)
    date_added = models.DateTimeField(default=datetime.today())

bar, created = Foo.objects.get_or_create(
    name = 'Alex', 
    date_added = some_datetime_obj
)

print created
# >> False

Responsive HTML5 Canvas and Processing.js

Thu, 13 Sep 2012 09:17:00 -0400

You know what sucks about html5? You have to specify the canvas element with exact width and height. Out of the box that means none of our fancy responsive/liquid grids will be able to get along well with the canvas. What’s worse is that when you start using something like Processing for graphics/designs, it too demands a specified width and height. When dealing with patterns, if you don’t it’s not fitting in the right dimensions it’s just going to look dumb and at that point just use a background image.

Fuck that. Here’s what I want to do:

Use a Processing sketch (using processing.js) to create a dynamically created background pattern
Use a liquid layout (in this example a modified Bootstrap responsive grid) that fits the user’s window size exactly
When the browser is resized the background pattern should fit precisely in the dimensions of it’s container

You can see the final result on the homepage of my new personal site http://astronautradio.com

Note: The rest of the site isn’t fully responsive yet, but the homepage is

~~Javascript~~ Jquery to the Rescue

This is how we are going to handle it with jquery. When the page loads we are going to resize all of the canvas elements on the page to the width and height of the window (or any percentage based width/height element). We will set a global variable for the width and height which will be used by the Processing sketch when it draws itself on the target canvas. We will either redraw an existing sketch or create a new one if it doesn’t exist.

# CoffeeScript
# Assuming all the dependancies are already loaded
resizeSketches = ->
    canvas = $ 'canvas'
    # We're using backticks here so that these get set globally
    `jswidth = $(window).width()`
    `jsheight = $(window).height()`
    canvas.attr 'width', jswidth
    canvas.attr 'height', jsheight
    sketch = Processing.getInstanceById 'id_of_the_canvas_el'
    if sketch == undefined
        # We need the html element and the sketch to run on it
        # I'm using a compiled processing js sketch called triangles
        new Proocessing $("#id_of_the_canvas_el")[0], triangles
    else
        # This is a function I wrote on the sketch to redraw itself
        # It's different than the builtin processing redraw method 
        sketch.restart()
    
# Somewhere in you code
$(window).on 'resize', resizeSketches

Call this when the page loads and it’s also triggered on the resize event. Lastly we need to update our Processing sketch to set it’s height and width based on the global javascript variables we specified.

Processing.js makes it super easy to share values between your other js code and the processing code. This is probably the most understated feature of processing.js, the ability to mash it up with javascript. In your sketch simply change the setup function by adding the following:

$p.size(jswidth, jsheight);

Easy! Now all we need is a method to restart the sketch which will wipe the existing one and draw it again. Something like…

function restart() {
  $p.background(255);
  // Reset the size of any other shapes that
  // depend on the canvas dimensions then
  $p.loop();

Now when the page is loaded and every time it’s resized, the canvas containing a processing sketch will set it’s dimensions and redraw itself. Now go forth and do not fear the fixed dimensions of the html5 canvas.

Another reason why I love Python

Sun, 09 Sep 2012 20:18:12 -0400

I didn’t get a chance to go see all of the amazing galleries in Brooklyn participating in #gobrooklynart this weekend. Browsing their list of artists revealed a lot of great pictures of their respective work. Naturally I wrote a script to parse through all of their artist pages to grab over 5000 pictures of awesomeness. I’m going to put the links I grabbed in a simple infinite scroll to digest the whole thing over this week.

Here’s the quick and dirty script I wrote in about 20 minutes:

# Get all the artists and all the links from Go! Brookly Art
# and put them in a text file for future use

from BeautifulSoup import BeautifulSoup
import requests
import json

pages = range(1, 115)

storage = open("gobrooklynart.txt", 'a')
for page_number in pages:
    url = "https://www.gobrooklynart.org/explore/artists?page=" + str(page_number) + "&neighborhoods=&media=Painting%2CPhotography%2CPerformance%2CVideo%2FFilm%2FSound%2CSculpture%2CPrint+Making%2FBook+Arts%2CIllustration%2CMixed+Media%2CTextile+Arts%2CDrawing%2CInstallation%2CDesign%2CFashion%2CCraft&accessibility=&order_by=&keyword="
    site = requests.get(url)
    html = BeautifulSoup(site.content)
    artists = list(set(html.find(id="search_results").findAll('a')))
    for artist in artists:
        artist_url = "https://www.gobrooklynart.org" + artist['href']
        artist_page = requests.get(artist_url).content
        artist_html = BeautifulSoup(artist_page)
        data = {}
        data['name'] = artist_html.find('h2', {'class':'display-name'}).text
        images = artist_html.find(id='profile-photos').findAll('img')
        data['pictures'] = [i['src'].replace("thumb", "standard") for i in images]
        try:
            data['homepage'] = artist_html.find(id='studio-website').findAll('a')[0]['href']
        except Exception, e:
            print e
            data['homepage'] = None
        print "Got artist %s" % data['name']
        storage.write(json.dumps(data) + ", ")
    print "On to page next page."

Now I have a txt document with a simple list of artists and pictures of their work. I actually forgot to put the extra “[ ” and “]”, but whatever you get the gist of it. Hooray!

(via FFFFOUND! | We are intensely aware of man as a machine and...

Tue, 04 Sep 2012 09:52:05 -0400

(via FFFFOUND! | We are intensely aware of man as a machine and the body as a mechanism - but does it float)

Drawing Triangles and Learning Processing part 1

Mon, 27 Aug 2012 14:47:00 -0400

I’ve spent about 20 hours now just drawing triangles using Processing. Processing is a programming language for drawing things with your computer. Recently, I’ve become more engrossed in the visual side of programming and exploring more of the intersection of the visual arts and technology.

As with learning anything, the best way for me is to actually do something with it and figure it out as I stumble along. I have a concept for a triangle pattern that inspired me and I want to recreate it in Processing so I can make it interactive (more on that in a future post).

Example (pardon the brutal colors):

How do you describe this pattern?

What I realized about visual programming is that I spend much more time thinking about how to explain the problem rather than figuring out how to code it. How do we explain this pattern? It’s a bunch of triangles, but if you just draw the same triangle over and over it won’t get that nice diamond pattern you see. You could break it down by line and you’ll find it’s two patterns that reflect each other. Or you could say that it has zig zags. After much pondering and help from a friend, we came up with an abstraction for explaining. Not as triangles, but as a square.

Imagine a two dimensional grid with a 3 x 3 dot grid. Each of the rows in the list of numbers represents a triangle. This is actually the pattern we see repeating that creates that diamond effect. I can now abstract the x,y coordinates with zeroes and ones that we will translate to real x and y points on the canvas.

There’s probably a mathematical formula here that could do this too, but that’s beyond my attention span. Instead we have a perfectly good abstraction of a single block that makes up an entire pattern. We can then replicate it on our canvas and achieve our pattern.

The Code

void setup()
{
  size(800, 600);
  frameRate(1);
  noStroke();
}

boolean over = false;  // If mouse over

void randomFill()
{
  fill(color( random(20, 255), 
              random(0, 255), 
              random(100, 255)));
}

void draw_triangle( int dimension, int x1_coord, 
                    int y1_coord, int x2_coord, int y2_coord, 
                    int x3_coord, int y3_coord)
{
  // Set the points of the triangle
  int x1 = dimension * x1_coord;
  int y1 = dimension * y1_coord;
  int x2 = dimension * x2_coord;
  int y2 = dimension * y2_coord;
  int x3 = dimension * x3_coord;
  int y3 = dimension * y3_coord;
  randomFill();
  triangle(x1, y1, x2, y2, x3, y3);
}

// Draw the 8 triangle pattern
void pattern(int x, int y, int dimension)
{
  pushMatrix();
  translate(x, y);
  // Look familiar? It's our triangle point abstraction
  draw_triangle(dimension, 0, 0, 0, 1, 1, 0);
  draw_triangle(dimension, 0, 1, 1, 0, 1, 1);
  draw_triangle(dimension, 1, 0, 1, 1, 2, 1);
  draw_triangle(dimension, 1, 0, 2, 1, 2, 0);
  draw_triangle(dimension, 0, 1, 0, 2, 1, 2);
  draw_triangle(dimension, 0, 1, 1, 1, 1, 2);
  draw_triangle(dimension, 1, 1, 1, 2, 2, 1);
  draw_triangle(dimension, 1, 2, 2, 1, 2, 2);
  popMatrix();
}

void draw()
{
  background(100);
  fill( 0, 121, 184 );
  int dimension = 100;
  for (int i = 0; i < 800; i = i+200) {
    pattern(0, i, dimension);
    pattern(200, i, dimension);
    pattern(400, i, dimension);
    pattern(600, i, dimension);
  }
}

This is just an early sketch to prove we can make this pattern using the square abstraction (it can be cleaned up a good amount too). It’s pretty simple when you break a pattern into larger chunks. We basically had to figure it out once, then replicate it a bunch of times to fill the canvas. Eventually, I’ll add add interactivity and a way to randomly generate the filling of each triangle. I’m also going to do some math to fit the number of columns and rows and the size of the triangles based on the canvas size. I plan on porting this over to processing.js for use on a web page. Cool stuff!

Introducing open-source, project based mentorship

Mon, 13 Aug 2012 14:31:50 -0400

Spoiler:

Project based mentorship is the future. I built http://mentorship.p2pu.org to connect people who want to learn (to code, design, whatever) or mentor by working on really cool projects.

A Promise

I’m a self-taught coder and I’m proud of it. I now make a living doing what I love: building cool shit. I would not have gotten to this point as fast as I did without the help of amazing mentors. In January I made a promise to help anyone who came to me looking for a mentor to help them learn to code. I’ve met some amazing people as a result and have tried my best to help make good on that promise.

I haven’t done enough. So I’m doing more because I believe in this.

I teamed up with P2PU, a non-profit, helping people learn from their peers online. I shared my story with them, what I believe in, and they gave me an amazing opportunity to team up with them in Berlin, where I built the P2PU Mentorship platform.

Project Based Mentorships is the Future of Learning

Unfortunately, just matching people with mentors based on what they are learning doesn’t work. I know this because I did it. By hand. Interviewing everyone, matching people together, sending intro emails, etc. It’s scary for most people. You agree to work for an indefinite amount of time with someone you don’t know.

A better way to do mentorships:

Create a project, something cool you want to work on
Collaborate with people looking to learn or mentor
Work on the project and learn along the way

Here’s an example:

I want to learn to build interactive websites
I create a project on P2PU Mentorship to build a cool site I’ve been dying to make
Mentors with the skills I want to learn join my project and help guide me
I work on the project and learn by doing with awesome mentors

Or if you are a mentor:

I’m working on something really cool, but I need help
I create a project on P2PU Mentorship to work with people who want to learn and I will help guide them kind of like an academic advisor
Learners join my project and we work on it together

Projects are the perfect opportunity to make mentorship happen online.

You’ll know what your getting into (you join project you’re interested in)
They are have a logical end (when the project is finished)
Learners get to produce something they are proud of, work with experienced people, learn new skills
Mentors benefit from more people working on their project, fresh ideas, and, of course, the joy of helping others

If you believe in online learning and helping others please join

If this sort of thing gets you excited, please join and help me pay it forward. Start a cool project and help make a difference.

Also if you are looking to learn Python and web development, the mentorship site itself is a project you can join led by yours truly. Check it out on github.

A simple real-time chat server using Clojure and Aleph

Sun, 05 Aug 2012 16:07:00 -0400

I’ve been exploring some asynchronous servers and have recently been enamored by Clojure, a modern lisp with brilliant built in concurrency primitives. In the Clojure world, your only real choice for an async server is Aleph. It’s built on top of an event-driven abstraction library called Lamina (from the same authors) which makes for a handy non-blocking server ready for realtime web apps powered by websockets, UDP, or TCP.

For this example I was mainly interested in a simple websockets web app with url routes using Compojure, an un-opinionated web framework in Clojure. The premise is simple; a public chatroom around a url (i.e /chat/hello has it’s own room and messages added there don’t show up on /chat/another and visa versa). I borrowed some of the code and patterns from another aleph example app you also might want to check out.

You can get the full working(ish) source here: https://github.com/alexkehayias/clojure-aleph-chat

(ns core.main
  (:use lamina.core
        aleph.http
        compojure.core
        (hiccup core page))
        ; This sets the correct file type for js includes used by hiccup 
        (ring.middleware resource file-info)
        core.views
        core.templates)
  (:require [compojure.route :as route])
  (:gen-class))

(defn page []
  "HTML page rendered using Hiccup. Includes the js we need for websockets."
  (html5
   [:head
    (include-css "/static/stylesheets/master.css")]
   [:body
    [:div.container
     [:div.row
      [:div.columns.twelve
        [:p [:h1#headline "Chat"]]
        [:form
          [:input#message {:type "text"}]
          [:input.nice.large.blue.button {:type "submit"}]]
	[:div#messages]]]]
    (include-js "http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js")
    (include-js "/static/javascripts/web_socket.js")
    (include-js "/static/javascripts/app.js")]))

(defn sync-app [request]
  "Rendered response of the chat page"
  {:status 200
   :headers {"content-type" "text/html"}
   :body (page)})

(def wrapped-sync-app
  "Wraps the response with static files"
  (-> sync-app
      (wrap-resource "public")
      (wrap-file-info)))

(defn chat-init [ch]
  "Initialize a new chat channel"
  (receive-all ch #(println "message: " %)))

(defn chat-handler [ch room]
  "Relays messages into a chat room. If it doesn't 
  exist create a new channel"
  (let [chat (named-channel room chat-init)]
    (siphon chat ch)
    (siphon ch chat)))

(defn chat [ch request]
  "View handler that handles a chat room. If it's not
  a websocket request then return a rendered html response."
  (let [params (:route-params request)
	room (:room params)]
    (if (:websocket request)
      (chat-handler ch room)
      (enqueue ch (wrapped-sync-app request)))))

(defroutes app-routes
  "Routes requests to their handler function. Captures dynamic variables."
  (GET ["/chat/:room", :room #"[a-zA-Z]+"] {}
       (wrap-aleph-handler chat))
  (GET ["/"] {} "Hello world!")
  ;;Route our public resources like css and js to the static url           
  (route/resources "/static")
  ;;Any url without a route handler will be served this response          
  (route/not-found "Page not found"))

(defn -main [& args]
  "Main thread for the server which starts an async server with 
  all the routes we specified and is websocket ready."
  (start-http-server (wrap-ring-handler app-routes)

So what’s happening here?

We’re taking a request to the URI /chat/<room> and creating a channel which other visitors to /chat/<room> subscribe to via a websocket. When a message is received to that URI it goes to the chat-handler, realizes it’s a websocket request, and adds the message to the named-channel that has that chatroom name. The message is pushed to all the subscribers of that channel (everyone on the page /chat/<room>) and the message is rendered on the page using javascript. There is only one channel that is persisted here that everyone is sharing.

All we need is some javascript to set up the websocket and handle messages. This is in the directory resources/public/javascripts:

// Note: this was generated from coffeescript and ƒ stands for function (lol emacs)
(ƒ() {
  $(ƒ() {
    window.socket = new WebSocket(window.location.href.replace("http://", "ws://"));
    socket.onopen = ƒ() {
      return console.log("socket opened");
    };
    socket.onmessage = ƒ(msg) {
      return $("#messages").append("<p>" + msg.data + "</p>");
    };
    return $("form").on("submit", ƒ(e) {
      e.preventDefault();
      socket.send($("#message").val());
      return $("#message").val("");
    });
  });

}).call(this);

There you have it, a simple, no-frills public chat room based on the page you’re on. This was just a test, but you can see how easy it is to set up a non-blocking async server with a little help from Aleph. Combined with the concurrency of Clojure itself, you can have an async, concurrent, functional programming love fest. Boners.

My pointless quest for the perfect web stack

Sat, 28 Jul 2012 11:37:39 -0400

Recently I’ve been on the blasphemes quest to find my ideal web stack.

What I Want

Modern web applications are pushing the envelope of the experiences that can be made over the internets. I use Python/Django/Postgres/Backbone/SASS at the moment and that’s all well and good. But what I want now is a concurrent language and a non-blocking web server to serve an API. Why? Because most of my web apps now are mostly client-side using lots of javascript and backbone. I want concurrent because I want speed for powerful algorithms I will theoretically write. I want an asnychronous web server because I’m tired of the Python -> Rabbitmq -> Celery dance I need to do to offload tasks into a queue.

I realize this list of stuff I want is ridiculous since I don’t actually know what I want to build with this theoretical stack. So keep that in mind that this is a pointless exercise, but one that I’ve been thinking about nonetheless. It’s also extremely narrow minded and not inclusive of all the stuff out there.

WINNER:

Not necessarily what I use now, but will use next.

Clojure/Aleph -> Mongo -> Backbone -> Handlebars -> SASS

BACKEND LANGUAGE:

1) Clojure

Concurrent
functional
lispy
fast
Still new, smaller community
Huge learning curve for me

2) Python

A caveman could write it
Tons of libraries
Proven
GIL sucks :(

3) Javascript

CoffeeScript makes it awesome
V8 makes it super fast
Possibility of one language for front and back end
single threaded

DB:

1) Mongo

Schema-less
easy to query
restful api (not sure how useful this is out of the box)
possible difficulties in scaling (anecdotal, but frequent)

2) Postgres/MySql

Relational
reliable
scalable
migrations can be a pain

WEB FRAMEWORK/SERVER:

1) Aleph (clojure)

The only real choice for non-blocking server in clojure
Websocket support
Works with Noir web framework
Socket.io not supported

2) Flask (python)

Super minimal
Simple to set up
It’s python

3) Express (node/javascript)

Lightweight
Readable
Large community and resources
Non-blocking
Easy Socket.io support

4) Django (python)

Batteries included
So many libraries makes rapid development easy
Get’s in your way after awhile (restrictive)
Synchronous unless you use Celery as a external task queue
Overkill for the kinds of web apps I write (Backbone, heavy client side)

FRONT END FRAMEWORK:

1) Backbone

Makes front-end feel like backend “mvc”
Namespacing keeps views separate
Non-prescriptive, use it how you want
Sometimes confusing because there is no right way to use it

TEMPLATING:

1) Handlebars

Separation of view logic (as much as possible)
Helpers let’s you write your own template tags
Multiple options for using templates

2) Moustache

Lacks helpers

CSS:

1) SASS

It’s just a thousand times better than writing cross-browser pure css

And there you have it. My ideal stack that is incredibly flexible and powerful to create awesome shit I haven’t thought of yet.

Backbone in the wild, lessons learned using backbone.js

Fri, 06 Jul 2012 11:14:31 -0400

This week I finished up a major redesign of Edorati which helps people discover articles with their friends. The reason I ended up going with a client-side MVC framework, Backbonejs, is because a huge piece of the existing web app was long javascript files with tons of jquery event handling. In short, it was a mess and difficult to build on top of. Although I didn’t set out to use backbone for everything, I ended up doing just that because it made modular development of client side code that much better. It made it feel like javascript was as real as the python/django that powers it.

Getting Your Head Around Client-Side MVC

Coming from a world of server-side templating, models, and views with Django, I had to start thinking of how that translates to backbone. Your backend will mostly be an API that serves data based on the requested URL and your business logic. Using backbone, my django project only serves one template for the whole app. Everything else is handled by backbone. That should give you an idea of how much heavy lifting backbone is able to do. I ended up stripping 70% of the view code I had written in django.

Oversimplified Explanation of Backbone

Here is my grossly oversimplified mental model of what backbone is.

Backbone organizes your front-end javascript code into an MVC style framework. There are 5 primary components to backbone:

Models: serialized data that describes an instance (similar to a django model without the database instructions)

Collections: a list of models (kind of like a queryset)

Views: a handler for a section of your app (similar to a django view, but typically more granular)

Router: routes a url to the appropriate view using a hashbang or pushState

Templates: Sticks you data into html. Not a part of backbone, but you will almost certainly use them. I personally use handlebars, but you can use whatever you want.

Backbone Patterns No One Tells You About

Going through the tutorials and reading other people’s code I’ve picked up on some basic patterns that are used by convention. Some of them were not immediately obvious when I started learning. Others I learned the hard way.

Granular Views - Consider a page that has a list of articles. With django you might create a view that grabs a queryset and outputs a template with a for loop to produce the list. You might then use jquery to handle events like the user editing that instance. With backbone you would set it up like this: A list view creates a collection of articles, for each model in the collection create an article view. Notice that there are two new pieces in there. The collection which is tied to a url which will call your api then automatically create a list of models from the received data and a view for an individual article. It was not obvious to me why in the world you would want a view for an individual article which might be 5 lines of code. The reason is for event handling. You’re basically setting the scope of events that apply to that code. This will help you avoid large, complicated views that try to handle all sorts of things for individual items. In the long run this leads to better separation of the different parts of your app and makes it much more manageable. Seems like more work up front, but it makes it easier in the end.
Zombies - I wish the documentation was more up front about this. Any non-trivial app typically creates and changes many views within a single session. YOU HAVE TO CLEAN THAT SHIT UP. If you don’t, your beautiful modular code will exhibit bizarre behavior as soon as you create a new view without first removing the old one. Events are delegated to elements. Even though you may have cleared the html, you have not removed the delegated events. Before you instantiate a new view that replaces another one, remove it and unbind all events from it. (this.remove() this.unbind()). It’s really hard to debug this, but the symptoms are usually things like the wrong ID being used, wrong model is changing etc. Zombies. For more info read this.
Requirejs - Non trivial apps will need to be separated into multiple files. The tutorials out there fail in explaining this. You will almost certainly be needing requirejs. Learn require well. Took me an embarrassing amount of time for me to get this.
One view per file - Related to the requirejs tip above, you only put 1 view in per file. Coming from django I just thought I would put a whole bunch of views in one file, but that’s not how it’s done with backbone using require. One view, model, collection, per file. Why? Because using require you need to return a class and it can’t handle you returning multiple classes.
Initializing an app - Using require, it’s not obvious how you “boot” the whole app. Create a view that will represent your app and create an initialize function to do what you need to start it up. In your main.js (used by require) create an instance of your app and voila, it will initialize. Typically you would initialize the router and any views that must be there no matter what. For Edorati that means the sidebar and router.
Router is your friend - Let your router handle the switching of views. Don’t try to stick the router logic into your views. Need to send people to a profile detail page? Create a route for it and use that url in the template. Don’t create an event handler in your view so that when someone clicks on it it creates a new view. Just use the router for what it does, route things. Doing this also means you’ll have re-usable links that can be used outside of the session. i.e. link that directly takes you to a user’s profile.

I’m sure I’ll think of more, but that’s it for now. Most of that list will not make sense until you dive into backbone creating your app. Tweet me if you have any questions @alexkehayias.

Write a script for non-trivial things you do more than once

Sun, 01 Jul 2012 13:20:00 -0400

I just wasted four hours migrating data semi-manually to a new server for a huge update to Edorati only to find that I made a mistake and all my work was rendered useless. What’s the problem here? The server? The data? No. Shit happens. Mistakes happen.

The moral of the story is if you have to do something more than once that involves a non trivial procedure just write a damn script.

Unfortunately, we have a bias against doing any additional work if we perceive to only need to do something a few times in an ideal scenario. The problem is that reality and ideal scenarios don’t usually co-exist. I should have been able to brush it off, mutter some choice swear words, and just re-run my script on a new server, but instead, I’ve been sitting hear bitching and complaining that the world as I know it is a soul-less, horrible, Bieber-filled place.

I ended up having to write the script anyway. It took me 20 minutes. It will still take a good 30 minutes for the script to run, but my involvement would go from 4 hours to about 5 minutes.

I’m going to go lay down on some train tracks for being so dumb.

My AngelHack Project, AlertBox, alerts for your social life

Mon, 25 Jun 2012 08:34:38 -0400

Had a great time with everyone this weekend at AngelHack, a startup hackathon with teams from all over the country. I worked on a little app called AlertBox to notify people of important updates from their social networks as soon as it happens. What if you could get alerted when your friend who just moved half way around the world is having a bad day? What would you do with that timely information?

What if we got the right information from our social networks just in time? I think the world could be a better place for it. It’s just a weekend hack, but the idea is powerful. What do you think?

Oh and it’s a python/django/backbone app. Enjoying backbone recently, will make an appearance in my portfolio much more often methinks.

6% Better than Twitter

Fri, 01 Jun 2012 14:06:46 -0400

The past two weeks I’ve been doing lots of pitching and am reminded just how important the question “what makes you unique?” is so important. If you’re answer is not a narrative about how you’re 10 times better than anything else the world has to solve a problem you will lose.

Incremental Improvements vs 10x Game Changers

When I was in DC last week I had a chance to see Dan Yates from opower present. He talked about the power of incremental improvements and how in the macro world they can represent a huge opportunity. For opower that means reducing power consumption for it’s clients (power companies) by 6%. Very different than a clean tech startup that is trying to replace power plants with solar/wind/whatever. That small change, in aggregate, is a huge improvement and they are able to capture some of the value they are creating (profit).

Then there are consumer web startups where an incremental improvement of 6% over anything is laughable. Imagine someone pitched their startup to you as “I’ve built a product that’s 6% better than Twitter.” You just can’t get excited about an incremental improvement when the web is this crowded. In one of Peter Thiel’s class note essays (go read all of them), Peter talks about dominating markets and becoming the last mover. Competition leads to incremental improvements. Domination comes from 10x game changers.

Don’t Compete, Dominate

There are plenty examples of incremental improvements leading to big impacts (Edison’s light bulb is an incremental improvement over what existed), but in the crowded world of web startups I feel that incremental won’t cut it. If you want to dominate you need to be a 10x improvement. Obviously if you’re not solving the right problem you can be 10x improvement on something no one cares about, so keep that in mind too. The point is to dominate not compete.

Lean Startups and Domination

I’m beginning to see how lean startup tactics are indoctrinating us towards incremental improvements. Maybe you have a big idea and vision of how you will change the world. You dissolve that into a testable series of assumptions and set off to prove it. As you go along and test, speak with people, and iterate you increment. You hone in on the actual problem and optimize different parts of your funnel. However, it’s too easy to fall in love with the tests, data, and incremental improvements. Sometimes the best edit is a rewrite (I think PG said that). Sometimes your pivot is just an incremental improvement. We need to think less about optimization and more about domination. You improved your signup conversion rate by 5%? Cool, but you still only have 100 users because no one cares, you’re not can’t miss, not novel, not 10x, just noise.

Where do we go from here

Maybe it’s time we start thinking about intelligent design rather than evolution (not in the education/political sense). If we start thinking about domination from day 1, how does that change our approach to product and distribution? There’s the danger of solving the wrong problem, but there’s also the problem of not solving the problem hard enough. I already know how to optimize, but I don’t know how to dominate. I’m going to work on that.

Advice for submitting Open Graph apps to Facebook

Tue, 08 May 2012 09:57:55 -0400

I recently had to submit an app with two Open Graph actions to Facebook for their blessing. It’s a clunky process where you need to have it fully working on externally available server (in my case a dev server), post the action to your account successfully, then tell them how to test it so that it meets their guidelines. Turn around time was about 2 days.

And then they rejected an action

I recently did a deeper integration with Facebook for Edorati to bring some social reader functionality so you can follow what your friends are reading more easily. One of the actions was approved right away, but the “read” action was rejected. OMG wut? It’s so far within their guidelines I could puke and it would still be kosher.

After thinking it through, I decided that my integration was flawless (I’m not actually as cocky as my writing presumes) and within guidelines. I resubmitted the same thing, no code changes, nothing. All I did was update my instructions for them to test it to specifically hit every point in their guidelines and it was approved.

Lesson

Avoid wasting time with the approval process by being very specific about your test case when submitting your app. Clearly hit on points such as global settings to turn the action on or off, per item control of the action (in my case, that’s a button for sharing an article on the article detail page), why it’s relevant to other people, and make your test case super easy to follow. Oh, it also helps if your code actually works ;-)

Here’s the Edorati submission (sanitized):

Log in to DEV_URL with username: USERNAME, password: PASSWORD. Go to ARTICLE_PAGE and click the “mark as read” button. This will perform the “read” Facebook open graph action. Clicking the button again will delete the post. Go to SETTINGS_PAGE to toggle the “read” action globally on or off. The read action is performed when a user reads an article which is a strong indicator that this is a relevant article that other people will want to read in their network. There is global control for this action as well as per article with the Mark as Read/Unread button on the article detail page.

How to make an interactive bookmarklet part 3

Tue, 01 May 2012 09:41:00 -0400

In the last two parts of this series we learned about the framework for building bookmarklets and how to use javascript with django templates for optimal maintainability. Now let’s look at how to make interactions work without needing any external javascript libraries.

Why not include Jquery

My guess is that all you really want are event bindings and for that you don’t need an entire library. Granted, lots of user’s have jquery cached in their browser already, but if not you have to include that which will make your bookmarklet slower. Every library you add will add more “weight” to the overall bookmarklet. Read the next paragraph before you decide you need the whole jquery library or you can make do with 6 lines of pure javascript.

Binding Events to Handlers

So you want to be fancy and make the bookmarklet change when the user clicks on something? Tough shit. Just kidding. The problem with event handling is that making it work cross-browser is tough. Jquery helps make writing js more cross-browser compatible and that’s a huge part of it’s overall value.

Here’s how we can add event handlers that will work across all browsers without Jquery. I can’t find where I originally found this from, but when I do I’ll add an attribution link.

    var bindEvent = function(elem, evt, cb) {
        //see if the addEventListener function exists on the element
        if ( elem.addEventListener ) {
            elem.addEventListener(evt,cb,false);
        //if addEventListener is not present, see if this is an IE browser
        } else if ( elem.attachEvent ) {
            //prefix the event type with "on"
            elem.attachEvent('on' + evt, function(){
                //use call to simulate addEventListener
                cb.call(event.srcElement,event);
            });
        }
    };

Now we have a simple way to make interactions happen within the bookmarklet itself. For example:

    // Remove the bookmarklet from the document assuming b is body and c is the bookmarklet
    var cancelDialog = function(e){
        b.removeChild(c);
    };
    bindEvent(cancelButton,"click", function(e){
        cancelDialog(this);
    });

All that without including the whole jquery library! That will keep your bookmarklet lightweight and fast.

Submitting a Form

Using our event binding snippet above we can create a simple handler for submitting a form free from cross-domain errors. Those of you familiar with jsonp will see that this is very similar, but done manually and in a simpler way.

    // Assuming we have the following in the script
    var d=document,
    s=d.createElement("scr"+"ipt");
    b.appendChild(s);
    
    // Bind a submit event to a button click
    var submitArticle = function(e){
        // get the values of any form data you need
        // URI encode the variables as GET parameters
        s.src = "http://www.urltoyourservice.com?var1=" + encodeURIComponent(variable1) + "&var2=" + encodeURIComponent(variable2);
    };
    
    bindEvent(submitButton,"click", function(e){
        submitArticle(this);
    });

This will make a GET request to your your server with the parameters of your form. We can now process that request and return a response just like we did to initialize the bookmarklet in the first place. All we need to do is respond back with javascript. You can do this numerous times. All your doing is manipulating the bookmarklet that is already a part of the document. Make sure that you clean up after yourself (see my earlier example of binding a cancel button to removing the bookmarklet from the document).

There you have it. An interactive bookmarklet that’s fast, maintainable, and lightweight. You can make changes to it at any time since the initialization happens server-side with familiar templates. You can make cross-domain requests without any errors. You can even bind events and handle them all without needing Jquery.

Enjoy!

How to make an interactive bookmarklet part 2

Wed, 18 Apr 2012 10:15:00 -0400

Let’s talk about how to generate a bookmarklet using server-side templating and pure javascript (no libraries!). In part 1 we discussed the basic framework for making a bookmarklet and performing requests directly from the client’s page. Now we’ll see how to set up the foundation for a bookmarklet UI and make it look good.

Building a Bookmarklet Template in Django

Let’s create a file called bookmarklet.js. In it we are going to add a self executing piece of javascript and leverage django templates to fill in the information we need. In here we are going to build the response using nothing but javascript and send it back to the user who requested the bookmarklet (remember all they did was request it to be initialized, see part 1 for more info). With django templates we can do some pretty handy things so that you don’t need to manually write out the template as a string in javascript. For example:

javascript:(function(){
    var d=document,
    {% comment %}Create the elements we need {% endcomment %}
    c=d.createElement("div"),
    b=d.body,
    l=d.location;
    {% comment %} server_scheme_and_netlog is my own tag which returns the base url of the website{% endcomment %}
    c.id="bookmarkletArticleAlertBox";
    {% comment %}This is where django templates get's awesome{% endcomment %}    
    {% comment %}Make sure the html rendered has no breaks{% endcomment %}    
    c.innerHTML = '{% spaceless %}{% include "bookmarklet_body.html" %}{% endspaceless %}';
    {% comment %}Add the elements to the dom{% endcomment %}
    b.appendChild(t);
    b.appendChild(c);

....snip....

What we did was construct a div that will hold our bookmarklet UI and using django templates (the {%include … %} part) we can template the html easily, inheriting the context of this view. Please make sure to add the “spaceless” tags here because the innerHTML will fail if there are line breaks and I’ll be damned if I have to add linebreaks to my template. Also be extremely picky with your quotes as that can also lead to disaster.

Now in our “bookmarklet_body.html” we can create our UI just as if it were any other django template (almost). For example:

{% load static %}
{% load url_tags %}
{% comment %}NOTE Make sure you ONLY use double quotes!{% endcomment %}    

<div id="bookmarkletAssistant">
    <img src="{% server_scheme_and_netloc %}{% get_static_prefix %}images/assistant.png" style="border: 2px solid #666666;"/><br/>    
    <img id="bookmarkletLoadingImage" src="{% server_scheme_and_netloc %}{% get_static_prefix %}images/bookmarklet-ajax-loader.gif" style="display:none;"/>
</div>
<div id="bookmarkletResponse">
    <p>Hey Chief, almost ready!</p>
    <div id="bookmarkletPictureSelector">
        <p>
            <label>Select Picture:</label><br/>
            <div id="bookmarkletImageContainer">
                <img id="bookmarkletCurrentImage" />
                <a id="bookmarkletSelectPrev">&laquo;</a><a id="bookmarkletSelectNext" >&raquo;</a>
            </div>
        </p>
    </div>
    <p id="bookmarkletImageCheckboxContainer"><input id="bookmarkletImageCheckbox" type="checkbox" style="margin-right: 5px;" />
    <label>No Picture</label></p>
    <p>
        <label>Editor&rsquo;s Note:</label><br/>
        <textarea id="bookmarkletTextArea" placeholder="Add your commentary on the article here. (optional)"></textarea>
    </p>
    <input id="bookmarkletSubmit" type="submit" value="Publish Article" />
    <a id="bookmarkletCancel">Cancel</a>
</div>

Notice how all the elements are contained within elements with IDs. This is extremely important so you can clean up after yourself and not pollute the dom. You will also need to do this so that subsequent interactions can find your bookmarklet and update the contents. BTW this is exactly what is used in Edorati.

Make sure that you have a view set up so that you know it is from your bookmarklet and can respond with your javascript template. When the user initializes the bookmarklet (see part 1 for reference) they will be requesting a url that you control. All you need to do is map that url to a view function and do the following:

    variables = {'some_var': 'blah blah'}
    resp = render_to_string('bookmarklet.js', variables)
    return HttpResponse(resp, mimetype="text/javascript")

This will make sure we are responding with javascript and not html. The javascript will be executed as soon as it is received by the user. It’s actually very fast and should feel close to instantaneous.

Making it Look Pretty

Plain unstyled html would be a terrible idea for a bookmarklet. Keep in mind that you are injecting onto another page where their stylesheet is unknown. That means we need to style and reset the css for the bookmarklet or it will not look consistent across sites. Here’s how:

    // in bookmarklet.js
    t=d.createElement("link"), 
    t.rel="stylesheet";
    t.media="screen, projections";
    t.type="text/css";
    t.href="{% server_scheme_and_netloc %}{% get_static_prefix %}stylesheets/bookmarklet.css";

Whoa, dynamically loading css just for our bookmarklet! Nothing fancy going on here, just using plain ol’ css to beautify our bookmarklet. Just make sure that you do not inadvertently style the rest of the elements on the page with your stylesheet. As mentioned before, add an ID for all major elements and when it comes to css, use the ID selector to style the element instead of the element selector or class. If you have to use classes just obfuscate the name so there is little possibility of you impacting the page i.e bookmarklet-textarea-field vs. textarea.

Next Week

Next week we’ll go over how to make the bookmarklet come alive with interaction through event binding and requests using about 15 lines of pure javascript and no libraries.

Adding a UUID Field to an Existing Django Model

Mon, 09 Apr 2012 09:47:29 -0400

It’s not immediately obvious how to add a UUID field to an existing Django model. Let’s say you have a database with some data in it about certain accounts, but you realize you need a way of obfuscating the pk of a model instance. UUID is a perfect candidate and a pretty popular implementation for Django comes from dcramer called django-uuidfield. Everything is fine and dandy until you try to add that field to an existing model that already has data, then try to auto migrate it using south.

How to add a UUID field to an existing model

The problem here is that we need to add a UUID value to all existing records, not just new ones. Peaking through the source code you will see what uuid method they are using and what the data should look like.

Add the UUID field to your model with blank=True, null=True, max_length=32, auto=True in models.py.
Run the schemamigration command with south and then open up the resulting migration file.
Edit your migration file with the following:

# You'll need to import the following
import uuid

# At the end of the forwards() function add the following
def forwards(self, orm):
    ...
    for a in MyModel.objects.all():
        a.uuid = u'' + str(uuid.uuid1().hex)
        a.save()

That will loop through all the existing instances of that model in your database and add a uuid to it as part of the migration. Make sure you test it first!

How to make an interactive bookmarklet part 1

Thu, 05 Apr 2012 10:11:00 -0400

Here’s how to put together a non-trivial interactive bookmarklet. For some reason there are not too many resources on the subject, but it has become a much more popular way of user’s interacting with a service without having to be on the site. Pinterest, Evernote, etc. all rely heavily on the ease of use of bookmarklets.

For Edorati, I needed to create a simple, fast way for user’s to publish an article to their online newspaper whenever they came across something they liked. It also had to look nice and allow them to select a picture thumbnail and add an editor’s note. Oh yeah it also needs to be upgradeable from the server side so that the user doesn’t have to reinstall it every time you want to add something new.

Why is it so hard?

Bookmarklets are especially hard for a couple reasons:

You don’t own the page the bookmarklet will deploy on
Cross-domain requests
Maintaining some level of security
Everything needs to be manipulated through javascript
Avoid using full js libraries like jquery

What’s the plan?

I ended up building a simple framework for bookmarklets and that uses django to process the request, handle errors, and provide responses. The bookmarklet is actually just a small piece of javascript code that gets executed as soon as it is clicked. It will tell my server that it’s ready to be initialized for a particular user. Then the server will make a response with ONLY javascript that is contained in a self executing function which will manipulate the target page the user is on. From there any interactions are a matter of building a request on the client side or returning a javascript response from the server-side. Sound simple? Not really…

How do I make cross-browser requests, do I use Jquery?

No Jquery! We’re going to avoid all libraries and roll our own. Adding big libraries will make your bookmarklet slow and you probably only need a couple of things from it. All we need in this case is a small workaround to make cross-browser requests right from the user’s page. This is the pattern:

javascript:(function(){
    var d=document,
    // Create a script element
    s=d.createElement("scr"+"ipt"),
    // Set the src to an endpoint on your server
    s.src = "http://www.urltoyourapi.com/?param1=something&param2=somethingelse";
})()

Now your server can respond with javascript to fill in that script element that was just created on the user’s page. By adding some GET parameters to the end of the URL you also have a way of communicating information about the user or the page. For example you may want to construct the url in a way that it passes the users obfuscated ID number so you can save information to that user’s account. This is all a bookmarklet is.

Making it future proof

To prevent your users from having to install the bookmarklet every time you want to make a change, you should not include any presentation or interactive logic in the bookmarklet. For Edorati, all I need to know from the bookmarklet is who is the user. All the bookmarklet needs to do is essentially tell the server that it wants to be initialized. Everything else should happen on the server side. That way you can make any changes you want to the look or the functionality without ever having to change the user’s bookmarklet that they installed.

Next post will discuss constructing templated javascript on the server side to make a bookmarklet interactive, clean, and safe…

Add fields to a Django serializer

Mon, 19 Mar 2012 19:23:36 -0400

Here’s a quick and dirty way to add fields to a serializer. I wanted to add in the absolute url to the json data that was being sent back during an ajax request. I wouldn’t recommend it for large querysets since you are essentially adding a json load and another dump, but for a single model instance I think it’s ok. The alternative would be to use a library like wadofstuff-django-serializer, but I really want to avoid a dependency for just a single purpose.

import simplejson as json
from django.core import serialize

# Somewhere in your view
# Note that I'm only serializing a single instance hence the tup
resp = serializers.serialize("json", (article,))

# Add absolute url to
resp_obj = json.loads(resp)
resp_obj[0]['fields']['get_absolute_url'] = article.get_absolute_url()
r = json.dumps(resp_obj)
# Then respond back as usual with your updated json object