Django model creation shortcut with kwargs
Let’s say you’ve got this model in Django that you need to create that has a lot of fields. You’ve already mapped out the data, but you think you need to write in each argument in the Model.objects.create method. Thankfully, these methods allow **kwargs to be passed in so you can do this:
# Remember that kwargs is just a dictionary that's mapping
# arguments to values (in this case model fields to values)
MyModel.objects.create(**some_dict)
# Or if we are updating it
MyModel.objects.update(**some_dict)
# Conveniently we could do something like this,
# assuming your field names are mapped exactly to the model names.
# Sometimes you don't want a model form so this works.
MyModel.objects.create(**request.POST)
# Oh, but I have a foreign key! No prob.
MyModel.objects.create(
fk_field = some_object,
**some_dict
)
Django Gotcha: Duplicate models when using get_or_create
When you’re saving some object to the db, there’s a gotcha with DateTimeFields that drove me nuts. It took me way too long to figure this out, but if you have a DateTimeField with the option auto_now_add, it will supersede a datetime object in a Model.objects.get_or_create call. This means that if you are tying to prevent duplicates based on date (say you’re saving some data from an API that has a timestamp) you will still get duplicates. It’s in the docs, but not the implications when using get_or_create. Check out this example:
# Given this simple model
class Foo(models.Model):
name = models.CharField(max_length=100)
date_added = models.DateTimeField(auto_now_add=True)
# This will always be true, even if an instance
# with this name and today's date already exists
bar, created = Foo.objects.get_or_create(
name = 'Alex',
date_added = some_datetime_obj
)
print created
# >> True
# The problem is, auto_now_add does some stuff that
# makes it uneditable, and fucks up my expectations
# when using it with get_or_create
# Here's the solution
class Foo(models.Model):
name = models.CharField(max_length=100)
date_added = models.DateTimeField(default=datetime.today())
bar, created = Foo.objects.get_or_create(
name = 'Alex',
date_added = some_datetime_obj
)
print created
# >> False
Another reason why I love Python
I didn’t get a chance to go see all of the amazing galleries in Brooklyn participating in #gobrooklynart this weekend. Browsing their list of artists revealed a lot of great pictures of their respective work. Naturally I wrote a script to parse through all of their artist pages to grab over 5000 pictures of awesomeness. I’m going to put the links I grabbed in a simple infinite scroll to digest the whole thing over this week.
Here’s the quick and dirty script I wrote in about 20 minutes:
# Get all the artists and all the links from Go! Brookly Art
# and put them in a text file for future use
from BeautifulSoup import BeautifulSoup
import requests
import json
pages = range(1, 115)
storage = open("gobrooklynart.txt", 'a')
for page_number in pages:
url = "https://www.gobrooklynart.org/explore/artists?page=" + str(page_number) + "&neighborhoods=&media=Painting%2CPhotography%2CPerformance%2CVideo%2FFilm%2FSound%2CSculpture%2CPrint+Making%2FBook+Arts%2CIllustration%2CMixed+Media%2CTextile+Arts%2CDrawing%2CInstallation%2CDesign%2CFashion%2CCraft&accessibility=&order_by=&keyword="
site = requests.get(url)
html = BeautifulSoup(site.content)
artists = list(set(html.find(id="search_results").findAll('a')))
for artist in artists:
artist_url = "https://www.gobrooklynart.org" + artist['href']
artist_page = requests.get(artist_url).content
artist_html = BeautifulSoup(artist_page)
data = {}
data['name'] = artist_html.find('h2', {'class':'display-name'}).text
images = artist_html.find(id='profile-photos').findAll('img')
data['pictures'] = [i['src'].replace("thumb", "standard") for i in images]
try:
data['homepage'] = artist_html.find(id='studio-website').findAll('a')[0]['href']
except Exception, e:
print e
data['homepage'] = None
print "Got artist %s" % data['name']
storage.write(json.dumps(data) + ", ")
print "On to page next page."
Now I have a txt document with a simple list of artists and pictures of their work. I actually forgot to put the extra “[ ” and “]”, but whatever you get the gist of it. Hooray!
My pointless quest for the perfect web stack
Recently I’ve been on the blasphemes quest to find my ideal web stack.
What I Want
Modern web applications are pushing the envelope of the experiences that can be made over the internets. I use Python/Django/Postgres/Backbone/SASS at the moment and that’s all well and good. But what I want now is a concurrent language and a non-blocking web server to serve an API. Why? Because most of my web apps now are mostly client-side using lots of javascript and backbone. I want concurrent because I want speed for powerful algorithms I will theoretically write. I want an asnychronous web server because I’m tired of the Python -> Rabbitmq -> Celery dance I need to do to offload tasks into a queue.
I realize this list of stuff I want is ridiculous since I don’t actually know what I want to build with this theoretical stack. So keep that in mind that this is a pointless exercise, but one that I’ve been thinking about nonetheless. It’s also extremely narrow minded and not inclusive of all the stuff out there.
WINNER:
Not necessarily what I use now, but will use next.
Clojure/Aleph -> Mongo -> Backbone -> Handlebars -> SASS
BACKEND LANGUAGE:
1) Clojure
- Concurrent
- functional
- lispy
- fast
- Still new, smaller community
- Huge learning curve for me
2) Python
- A caveman could write it
- Tons of libraries
- Proven
- GIL sucks :(
3) Javascript
- CoffeeScript makes it awesome
- V8 makes it super fast
- Possibility of one language for front and back end
- single threaded
DB:
1) Mongo
- Schema-less
- easy to query
- restful api (not sure how useful this is out of the box)
- possible difficulties in scaling (anecdotal, but frequent)
2) Postgres/MySql
- Relational
- reliable
- scalable
- migrations can be a pain
WEB FRAMEWORK/SERVER:
1) Aleph (clojure)
- The only real choice for non-blocking server in clojure
- Websocket support
- Works with Noir web framework
- Socket.io not supported
2) Flask (python)
- Super minimal
- Simple to set up
- It’s python
3) Express (node/javascript)
- Lightweight
- Readable
- Large community and resources
- Non-blocking
- Easy Socket.io support
4) Django (python)
- Batteries included
- So many libraries makes rapid development easy
- Get’s in your way after awhile (restrictive)
- Synchronous unless you use Celery as a external task queue
- Overkill for the kinds of web apps I write (Backbone, heavy client side)
FRONT END FRAMEWORK:
1) Backbone
- Makes front-end feel like backend “mvc”
- Namespacing keeps views separate
- Non-prescriptive, use it how you want
- Sometimes confusing because there is no right way to use it
TEMPLATING:
1) Handlebars
- Separation of view logic (as much as possible)
- Helpers let’s you write your own template tags
- Multiple options for using templates
2) Moustache
- Lacks helpers
CSS:
1) SASS
- It’s just a thousand times better than writing cross-browser pure css
And there you have it. My ideal stack that is incredibly flexible and powerful to create awesome shit I haven’t thought of yet.
How to make an interactive bookmarklet part 3
In the last two parts of this series we learned about the framework for building bookmarklets and how to use javascript with django templates for optimal maintainability. Now let’s look at how to make interactions work without needing any external javascript libraries.
Why not include Jquery
My guess is that all you really want are event bindings and for that you don’t need an entire library. Granted, lots of user’s have jquery cached in their browser already, but if not you have to include that which will make your bookmarklet slower. Every library you add will add more “weight” to the overall bookmarklet. Read the next paragraph before you decide you need the whole jquery library or you can make do with 6 lines of pure javascript.
Binding Events to Handlers
So you want to be fancy and make the bookmarklet change when the user clicks on something? Tough shit. Just kidding. The problem with event handling is that making it work cross-browser is tough. Jquery helps make writing js more cross-browser compatible and that’s a huge part of it’s overall value.
Here’s how we can add event handlers that will work across all browsers without Jquery. I can’t find where I originally found this from, but when I do I’ll add an attribution link.
var bindEvent = function(elem, evt, cb) {
//see if the addEventListener function exists on the element
if ( elem.addEventListener ) {
elem.addEventListener(evt,cb,false);
//if addEventListener is not present, see if this is an IE browser
} else if ( elem.attachEvent ) {
//prefix the event type with "on"
elem.attachEvent('on' + evt, function(){
//use call to simulate addEventListener
cb.call(event.srcElement,event);
});
}
};
Now we have a simple way to make interactions happen within the bookmarklet itself. For example:
// Remove the bookmarklet from the document assuming b is body and c is the bookmarklet
var cancelDialog = function(e){
b.removeChild(c);
};
bindEvent(cancelButton,"click", function(e){
cancelDialog(this);
});
All that without including the whole jquery library! That will keep your bookmarklet lightweight and fast.
Submitting a Form
Using our event binding snippet above we can create a simple handler for submitting a form free from cross-domain errors. Those of you familiar with jsonp will see that this is very similar, but done manually and in a simpler way.
// Assuming we have the following in the script
var d=document,
s=d.createElement("scr"+"ipt");
b.appendChild(s);
// Bind a submit event to a button click
var submitArticle = function(e){
// get the values of any form data you need
// URI encode the variables as GET parameters
s.src = "http://www.urltoyourservice.com?var1=" + encodeURIComponent(variable1) + "&var2=" + encodeURIComponent(variable2);
};
bindEvent(submitButton,"click", function(e){
submitArticle(this);
});
This will make a GET request to your your server with the parameters of your form. We can now process that request and return a response just like we did to initialize the bookmarklet in the first place. All we need to do is respond back with javascript. You can do this numerous times. All your doing is manipulating the bookmarklet that is already a part of the document. Make sure that you clean up after yourself (see my earlier example of binding a cancel button to removing the bookmarklet from the document).
There you have it. An interactive bookmarklet that’s fast, maintainable, and lightweight. You can make changes to it at any time since the initialization happens server-side with familiar templates. You can make cross-domain requests without any errors. You can even bind events and handle them all without needing Jquery.
Enjoy!
Adding a UUID Field to an Existing Django Model
It’s not immediately obvious how to add a UUID field to an existing Django model. Let’s say you have a database with some data in it about certain accounts, but you realize you need a way of obfuscating the pk of a model instance. UUID is a perfect candidate and a pretty popular implementation for Django comes from dcramer called django-uuidfield. Everything is fine and dandy until you try to add that field to an existing model that already has data, then try to auto migrate it using south.
How to add a UUID field to an existing model
The problem here is that we need to add a UUID value to all existing records, not just new ones. Peaking through the source code you will see what uuid method they are using and what the data should look like.
- Add the UUID field to your model with blank=True, null=True, max_length=32, auto=True in models.py.
- Run the schemamigration command with south and then open up the resulting migration file.
- Edit your migration file with the following:
# You'll need to import the following
import uuid
# At the end of the forwards() function add the following
def forwards(self, orm):
...
for a in MyModel.objects.all():
a.uuid = u'' + str(uuid.uuid1().hex)
a.save()
That will loop through all the existing instances of that model in your database and add a uuid to it as part of the migration. Make sure you test it first!
Cohort Analysis in Django
I wrote a simple reusable app for doing cohort analysis in Django. Cohort analysis is an incredible tool for deflating your ego. Since it compares apples to apples you will see if you are actually helping, hurting or, more than likely, making no impact at all on your users with the iterations you’ve been making on you’re awesome web app.
What is Cohort Analysis?
Cohort analysis involves segmenting your users into smaller groups so you can compare measurements on them. For example, a cohort could be the user’s that signed up for your app in a week. So all the people that joined last week is a group, everyone who joined two weeks ago are in a group, etc. Now we can analyze actions that the cohort has taken in relation to the week they joined. For example “Users who joined the week of 2/1 performed 10% more action x in their first week than the users who joined in 1/25.” You can then think about the changes you made and correlate it back to the results. This prevents things like traffic spikes or that PR you did from messing up your numbers. You’ll be able to see over time if you are actually getting any better rather than looking at aggregate statistics of the activity of all your users. If you try hard enough with any aggregate stat you can make it look like a hockey stick. But that’s bullshit because you have no idea if the changes you made are the cause of that improvement when you look in the aggregate.
Cohort Analysis in Django
So how do we do this in django? I wrote a very simple, reausable app called django-cohorts. The objective is to set up a simple framework for running cohort analysis defined by the segments (cohorts) you want and the metrics you want in a dashboard that lives in the admin section. It’s raw right now, but it’s a start.
How does it work?
I’ve done the base cohorts by week, which is pretty common, but you can override that. In metrics.py, create a class with functions that will return a percentage. It will send you the users and you do some db lookup for certain actions the user took. After you’ve hooked it up to your url conf, you can then see the analysis with a dropdown of the metrics that are available.
Looking for Contributors
If anyone wants to help make this more sophisticated please let me know. Feel free to fork it on GitHub.
A Note about Cohort Analysis
Since cohort analysis typically looks at usage over time, you need to have some data for it to be useful. You can’t identify patterns that are significant without at least a few weeks of data. Just saying.
This is why I love Flask
from flask import Flask
from flask import render_template, url_for
app = Flask(__name__)
@app.route('/')
def homepage():
return render_template('index.html')
if __name__ == '__main__':
app.run(debug=True)
url_for('static', filename='/stylesheets/compass_foundation.css')
Cool! This is a single page app using Flask. Oh and you can run a test server with “python <name_of_file>.py”. Don’t need no full-blown feature heavy framework, no stinkin ORM, no form handling, authentication, etc for this simple project. Somehow I feel cleaner using Flask… How much of Django am I actually utilizing? I’m beginning to see the light about python web microframeworks.
Using PJAX to Speed Up a Django Web Application
Last week I started working on a new side project with my business partner @artiepatel called TimeoutDebate, an interactive sports blog and game like ESPN’s PTI (note, it’s not fully useable on iOS browsers yet). I’ve checked out PJAX and thought it would fit perfectly for this kind of content based website. Basically, it makes everything appear really snappy because all of the pages you view are loaded right into the parent page without a full page load. Sounds like ajax right? Yes, except that it pushes the actual url and all of it’s context just like a full page load sans the page load. Confused yet? Take a look at http://www.timeoutdebate.com to see it in action. Notice the lack of page loading, but the urls changing.
PJAX and Django
The good thing about PJAX is that it is really simple to get up and running… for simple stuff. There is a library called django-pjax which provides the foundation for what you need on the server-side. The rest is handled in your templates. It’s tough to know if it’s working (especially when running it locally on the devserver) so open up firebug so you can see the ajax requests being made with “_pjax=true”.
Template Re-use
Much to my horror, it looked like I needed to duplicate templates for pjax and non pjax requests. This is by design since a user can still navigate directly to a url without using a pjax request. You don’t want to leave them in the cold, but does that mean you have to maintain nearly identical templates? Here’s what I found to be effective at cutting down on the template duplication and maximize re-use.
This is my debate detail page that get’s loaded on non-pjax requests e.g. someone navigates directly to /debates/2/something-something and not via a link on the homepage.
{% extends 'index.html' %}
{% block content %}
{% comment %}Automatically re-use the pjax version of the
template to avoid duplication. This template shown if the user
reloads the browser or links direct to the content.{% endcomment %}
{% include 'debate-pjax.html' %}
{% endblock %}
I simplified this a little bit for the sake of example. I actually needed to include a script for reloading the Facebook comments section. However, this will still allow you to reuse your pjax pages even on non-pjax requests. WIN!
I also do this on the homepage to avoid the need for a index-pjax template. It’s not your typical homepage since we show you the latest sports debates instead of some static homepage bullshit.
{% block content %}
{% include 'debate-pjax.html' with debate=latest_debate %}
{% endblock %}
This is in my homepage template. When someone navigates right to timeoutdebate.com they are shown the latest news. I pass along a debate object that I rename to debate using the handy ’with’ statement inside an include tag so I can reuse the debate-pjax template again.
How the rest of TimeoutDebate works with PJAX
There is actually a lot going on in the javascript including ajaxified voting functionality, Facebook comments, and sidebar navigation. The nice thing about PJAX is that these pages are handled just like a normal view. You don’t need to do much crazy manipulation using javascript and ajax calls since you django view handles most of the variables you need right in the template.
Voting
The django view is what controls whether you can vote or not and the message returned in the modal window that pops up after a user clicks it. An ajax call posts the vote with the opinion ID and whether it’s positive or negative.
Sidebar nav
This was actually one of the trickier parts of the whole pjax experience. I needed to know what item the user was looking at and update the side nav to highlight the matching item. In the debate template I have an element that identifies what debate is loaded in the main div. I then match it to the sidebar nav item name and highlight it. I purposely have the sidebar outside of the main div where the pjax page is being pushed so that I don’t need to reload the entire nav every time someone looks at a debate.
Conclusion
PJAX has a ton of potential, but involved a lot of trial and error on my part to get it working in a dynamic web application. If the pjax request fails due to some other javascript bug there is the potential that jquery will hang the whole browser (which is TERRIBLE). There is also issues with iOS browsers, but that’s a topic for some other time. I still have a bunch of work to do to speed up TimeoutDebate, but the foundation is there. Check it out if you haven’t already!
Learning to Code - Choosing a Language
Based on all the feedback I’ve been getting from interviewing all the Developer Mentor Program, the first problem that comes up when someone decides they want to learn to code is often “Where do I begin?” This should help all the #learntocode2012 people narrow down what you want to learn.
Backend vs Frontend
Web programming is divided up into two sides (you may have heard these terms before); front end and back end. The frontend includes everything the user sees and interacts with. The backend provides all of the functionality, storage, and business logic for the application/website. If you really want to be able to build things from scratch by yourself you’ll need to learn both sides. That may sound daunting, so my advice is to start small and break it into manageable chunks.
Choosing a language
Now that we know generally what the frontend and backend are we can start deciding what language(s) to start learning.
When I read the Backend vs Frontend paragraph I thought…
1) “Frontend! I want to be able to make things pretty and useable”
2) “Backend! I want to create awesome features to solve problems”
3) “Dammit I want to learn both now get on with it!”
If you answered (1) or (3)…
Frontend languages for modern websites and web applications:
html
you probably took an introductory course to html at some point in your life. If not, right click on this webpage and select “view source.” That’s html and it’s what gives a webpage something to look at.
css
this makes said webpages pretty. It styles all of the elements on the page and determines the look and feel (sexiness)
javascript
This is what drives those subtle interactions you probably never realized. It’s what allows you to make things feel “snappy” and “fast” and prevent lots of page loads. It can do a ton of other things, but the reason it’s cool is that it’s baked into all browsers.
*I didn’t mention html5 as a separate item because it’s still part of html. Personally, I don’t use many html5 features simply because it does not support older browsers and majority of the interactions I need to create can be done in javascript. Not everyone has the latest version of Chrome you internet hipster!
If you answered (2) or (3)…
Backend languages for modern websites and web applications:
Python
It’s no secret, I love me some Python. It was my first scripting language and I find it to legible, expressive, and relatively easy to grasp. It has an incredible community and a well developed web framework called Django.
Who uses python?
- Youtube
Ruby
Ruby has become the backend language of choice for a large amount of the web applications you see out there today. People who want to build web application fast and furious (many startups) use the Rails (you may have heard of RoR or Ruby on Rails) web framework. It does a lot of magic behind the scenes so you spend less time configuring and more time building.
Who uses Ruby on Rails?
- Every startup on the east coast ;-) (no but seriously though, there is a huge demand for ruby developers in the startup community)
PHP
PHP has been around for awhile. That’s about all I’m qualified to say about it. It’s a proven backend that has been used many times over, but recently I’ve noticed that it has fallen out of favor in the startup community (which is largely python, ruby).
Who uses PHP?
- Facebook (you may have heard of them)
Javascript
What!? I though this was a frontend language! Well that’s half true. With the arrival of NodeJS, good ol’ javascript has a new job of playing backend. That means you can write backend functionality in javascript (which many of you may already know).
Who uses Node?
- The good folks at Shelby.tv
- There’s definitely others, feel free to leave me a comment about it
Which one should I go with?
If you want my advice (and you should), go with the one that the people around you are using. If you have some buddies that know Python, go with Python. I already pointed out the languages I would recommend (notice they are all open source, scalable, community-driven, and awesome), but having access to a community of local developers is really important. Why did I learn python? Because my brother and Jordan know python. Simple as that. Just make sure you pick from the ones I recommend above.
* Please no fanboy comments about which language you think is better. That’s cool bro, go code in that.
One More Thing
Stop worrying about which language to learn and just pick one. Using my nifty guide from this post, whichever language you pick, you can’t go wrong. You can build amazing things in any language. It’s the craftsman, not the tools.
