Showing posts with label GAE. Show all posts
Showing posts with label GAE. Show all posts

Wednesday, 30 December 2009

Google App Engine: paging with inequality filters

I recently implemented paging for my Google App Engine app, Braineos, ordering results using a unique inequality filter. My implementation is based on Joe Gregorio's Paging through large datasets tutorial, but tweaked in order to overcome apparent flaws in Gregorio's implementation and to meet the search result ordering requirements of my application.

Paginating results has important performance implications. While GAE limits results sets to 1000 and it's unlikely that a user will ever require more than a 1000 results, fetching up to the maximum number of results is less optimal that fetching a smaller defined limit with each query: pagination is included in Google's Best practices for writing scalable applications.

Gregorio describes several means of ordering paginated results, and the paging on a unique property approach was most appropriate for the needs of my application: I wished to order user generated content first by the number of positive 'votes' the content received and second by the most recently created content.  This approach uses shard counters to enable the creation of a unique key as well as quick updates to the datastore. Gregorio's Contributor class and unique 'when' property creation were added to my code base (the when property added to the model object representing the user generated content)[1]:

class Contributor (db.Model):
count = db.IntegerProperty(default=0)

def _unique_user(user):
email = user.email()

def txn():
contributor = Contributor.get_by_key_name(email)
if contributor == None:
contributor = Contributor(key_name=email) 
contributor.count += 1
contributor.put()
return contributor.count

count = db.run_in_transaction(txn)

return hashlib.md5(email + '|' + str(count)).hexdigest()

def whenfromcreated(created):
return created.isoformat()[0:19] + '|' + _unique_user(users.get_current_user())

My search module then used the 'when' property when ordering results sets:

def find(self, searchTerm, pagesize=configuration.Configuration.PAGESIZE):
searchTerm = searchTerm.split(',')
searchTerm = [term.strip().lower() for term in searchTerm]       
query = UGC.all()
for term in searchTerm:
query.filter('searchTerms =', term)
ugcs = query.order('-when').fetch(pagesize + 1)
  return ugcs

def findNext(self, searchTerm, next, pagesize=configuration.Configuration.PAGESIZE):
searchTerm = searchTerm.split(',')
searchTerm = [term.strip().lower() for term in searchTerm]    
query = UGC.all()
for term in searchTerm:
query.filter('searchTerms =', term)
query.filter('when <= ', next).order('-when')
ugcs = query.fetch(pagesize + 1)
return ugcs

This implementation fitted with the app's existing shard counting implementation and provided an excellent starting point. I came up against two issues. First, appending the count to the user's email before digesting with MD5 resulted in unreliable desc ordering. Second, because the 'when' property is used as an inequality filter, ordering must first be on that property (so I couldn't first order my results on 'votes' and then 'when').

I worked around the first issue by appending the count value after the user's email was digested:

return hashlib.md5(email).hexdigest() + '|' + str(count)


I worked around the second issue by prepending the number of votes to the 'when' property:

def whenfromcreated(created, votes=0):
return votes + '|' + created.isoformat()[0:19] + '|' + _unique_user(users.get_current_user())

This approach appeared to behave as expected, however my initial unit test used a sample of less than 10 UGC objects and it became apparent that once a user's contributions exceeded nine, desc ordering would be inaccurate: a unique user string ending in '|9' is deemed greater than a string ending in '|10'. Happily a fellow GAE app developer encountered a similar problem when attempting to use int values in a str comparison and suggested converting the int values to hex to allow alphabetical comparison.[2] I altered the code to wrap votes and count in the hex() factory function and my unit test, which seeks to order 10 items into two pages, is now passing.

There is significant discussion around pagination in GAE and while my implementation seems to stick pretty close to Google's reference implementation, if others have devised more elegant solutions, I'd be interested to know more - comment below!

[1] Gregorio's full code sample is available here.
[2] See Paging - ordered by a signed integer

Wednesday, 17 June 2009

Pretty URLs: Routes in Google App Engine

Pretty URLs are an essential element of web applications. Pretty URLs are helpful to users, influential for SEO and implementations may enhance the beauty (and so the extensibility and maintainability) of a code base. This is superbly achieved in Google App Engine using Rodrigo Moraes' App Engine recipe for URL mapping using Routes.

I discussed in a previous post how I used a basic implementation of GAE's WSGIApplication for Pretty URL mapping. While this approach served my initial needs, it required a lot of extra logic to read params from incoming URLs and did not elegantly handle URLs invoked in an AJAX call (i.e. logic was need in the handler to interpret the URL and delegate to the appropriate action). My goal was to implement a Pretty URL mapping scheme that allowed me to:

-configure URL mappings in one place only
-gracefully pass params on URLs
-invoke actions contained in my handlers

Happily, I stumbled upon Moraes' recipe; it was a snap to follow the very clear directions to import Routes and override the WSGIApplication. Further support from the Routes documentation allowed me to quickly and easily meet all my Pretty URL goals.

All URL mappings are configured in routing.py. Params passed on URLs are discretely mapped and retrieved. For instance this mapping accepts a param named 'id' and hits a handler in views.py called 'MyHandler':

map.connect('handler/:id', controller = 'views:MyHandler')

In views.py, 'id' is picked up as an argument of the get() method:


class MyHandler(ParentHandler):
def get(self, id): ...


How cool is that?! Crisp and clean and no caffeine. It gets better.

I can also map to a handler and an action. This is particularly handy for my api.py, which accepts AJAX requests. With this mapping the correct handler's action is invoked without any messy delegation at the top of my api.py:

map.connect('api/handler/action', controller = 'api:MyHandler', action = 'action')

How great is this? Pretty URLs and cleaner code. Yes, I've added the Routes resources and Moraes' overridden WSGIApplication, but the separation of mapping logic from other aspects of the application pleases me and I believe makes for a more robust code base. Not to mention, I learned a new thing! Thanks Mr Moraes, my URLs got pulchriTUDE!

Monday, 23 March 2009

Pretty URLs in Google App Engine

Today I implemented so-called 'pretty urls' for my Google App Engine application. While my URL structure was reasonably clean before this work, I was inspired to make improvements after attending ThinkVisibility and particularly after talks by Rob Lee and Tom Smith. While it wasn't too painful to track down information on how I might achieve Pretty URLs in GAE, it did take a little trial and error to get running, so I thought it worth a post. Indeed, if readers have suggestions for a more elegant solution, I welcome comments.

What is a Pretty URL? A Pretty URL is a URL designed to be 'hackable', or human-readable. Ideally, users should be able to intuit a Pretty URL address, and likewise a Pretty URL might serve as a 'bread crumb' or 'sign post' indicating to the user where s/he is in the application. Compare, for example, a less than fetching URL (a) with a more pleasing example (b):

a) mysite.com/mycategory?myparam=an_entity_id
b) mysite.com/mycategory/an_entity_id

Example 'b' is just the sort of URL I desired; purdy, ain't she?

I chose to use my WSGIApplication's constructor to point Pretty URLs at appropriate Handlers:

application = webapp.WSGIApplication([('/category/.*', CategoryHandler)], debug=True)

The 'category' is associated with a particular Handler, while the '.*' regex matches the UID of the entity to be retrieved, which is passed on the URL. In the Handler's get function I retrieve the path instance variable include on the Request object, which returns a substring of the URL between the host name and query parameters. I then substring the path string in order to retrieve the key of the entity requested by the user:

path = str(self.request.path)
key = path[6:len(path)]
if key:
#Retrieve the entity and shoot back a response


This appears to be working well and as expected; I'm certainly blushing every time I lay eyes on one of my app's sweet little URLs.

(Readers may wish to view Google's doc on URL config; note if you try copying and pasting the syntax in their example you'll get exceptions, hence the trial and error aspect of my implementation.)