From time to time one needs to fetch a random row from a table in a db.

Solution

Take one

The “obvious” solution is to order by RAND() and get the first:

SELECT * FROM users ORDER BY RAND() LIMIT 1; /* Don't do that */

It does the job, problem solved!

Well, no, not exactly! While it works, the performance will be awful for any table of significant size.

Take two

There is another, simple, but efficient, method to get a random row:

1. Fetch the number of rows using COUNT().

   SELECT COUNT(*) FROM users;

2. Get a random positive number, that is less than (but not equal) to that count.
3. Select the row using OFFSET (on MySQL – there are ways to do the same on all major RDBMS)

   SELECT * FROM users LIMIT 1 OFFSET :rand

   where :rand is the number you computed.

While this uses two queries – both of them are very efficient.

Here is the concrete code on how to do that in Django (using a custom manager):

class UsersManager(models.Manager):
    def random(self):
        count = self.aggregate(count=Count('id'))['count']
        random_index = randint(0, count - 1)
        return self.all()[random_index]
 
class User(models.Model):
    objects = UsersManager()
    # ... [fields] ...

Usage is as simple as

User.objects.random()

Just for the record – the Django way to shoot yourself in the foot do ORDER BY RAND() is

User.objects.order_by('?')  # If you do that in production - you may as well forget about sleep!

Proof

Here is the evidence that doing ORDER BY RAND() is very slow.

For the test we will have a simple Django model and a few of scripts – one that populates the database with bulk data, one that fetches a user using ORDER BY RAND() and one that fetches it using a second query. We populate the db with one million records and get a random one.

Here are the results (done using Python 2.6.4 and MySQL 5.1/SQLite 3.6.16 on Ubuntu 9.10):

Using the naive method by doing ORDER BY RAND():

$ time python naive.py
#       MySQL        SQLite
real	0m4.519s     0m2.061s # seconds?!?! 
user	0m0.152s     0m1.952s
sys	0m0.028s     0m0.056s

(executing the query directly results in times close to those).

Using the smart method with two queries:

$ time python smart.py 
#       MySQL        SQLite
real	0m0.367s     0m0.339s # much better
user	0m0.156s     0m0.284s
sys	0m0.032s     0m0.052s

The more records you add to the table, the slower the naive method will get, while the smart one will run at about the same speed.

If you need to get a random record out of a filtered set (using WHERE) is’s basically the same. You just need to add the WHERE clause to both queries.

Details

Below is the code that is used to make the benchmark.

This is the models.py file of a simple Django app.

from random import randint
from django.db import models
from django.db.models import Count
 
class UserManager(models.Manager):
    def random(self):
        count = self.aggregate(ids=Count('id'))['ids']
        random_index = randint(0, count - 1)
        return self.all()[random_index]
 
    def random_naive(self):
        return self.all().order_by('?')[0]
 
class User(models.Model):
    username = models.CharField(max_length=128)
    password = models.CharField(max_length=128)
 
    objects = UserManager()

The bluk.py script that will fill the db with junk.

import os
import settings
 
os.environ['DJANGO_SETTINGS_MODULE'] = os.path.join(
    os.path.dirname(__file__), 'settings')
 
from users.models import User
if __name__ == '__main__':
    for i in range(0, 1000000):
        u = User(username="user{0}".format(i), password="pass{0}".format(i))
        u.save()

This is naive.py – the script that uses the slow method.

import os
import settings
 
os.environ['DJANGO_SETTINGS_MODULE'] = os.path.join(
    os.path.dirname(__file__), 'settings')
 
from users.models import User
 
if __name__ == '__main__':
    u = User.objects.random_naive()
    print(u.username)

And smart.py that uses an additional query.

import os
import settings
 
os.environ['DJANGO_SETTINGS_MODULE'] = os.path.join(
    os.path.dirname(__file__), 'settings')
 
from users.models import User
 
if __name__ == '__main__':
    u = User.objects.random()
    print(u.username)

What is Null?

Here is an explanation from wikipedia.

Null is a special pointer value (or other kind of object reference) used to signify that a pointer intentionally does not point to (or refer to) an object.

From an object oriented language point of view Null is a a value, whose type is a subclass of every other type in the system. It’s always at the bottom of the hierarchy. Because of this and the Liskov substitution principle Null can be used everywhere normally other type would be used. Have a method, that is marked to return string? It can return Null.

What is the problem with Null

The problem with it is that every method can return Null. And you, as a consumer of that method, need to check every time if the return value is Null. If you don’t – you (usually) get a Null pointer exception and your program crashes.

Now, exceptions are a very good tool – but particularly the Null pointer exception is a bad thing. Why? Because if it bubbles more than one level up it’s a sign of a leaky abstraction. Let me back up this claim:

First, what does a Null pointer exception mean? Let’s have a look at the documentation of it in Java:

Thrown when an application attempts to use null in a case where an object is required. These include:

• Calling the instance method of a null object.
• Accessing or modifying the field of a null object.
• Taking the length of null as if it were an array.
• Accessing or modifying the slots of null as if it were an array.
• Throwing null as if it were a Throwable value.

Basically it says that an object is expected, but no such is provided. Well, this is an implementation detail. In your business logic I hardly believe that you will be talking about references and such. We usually talk about real world object like people, accounts, etc. Any method that throws a Null pointer exception unveils parts of its implementation.

How many times has happened to you to forget to fill a field in some form and get a Null pointer exception.
I forgot to write my email – what does that Null thing has to do with it?!?

A method should never throw Null pointer exceptions (I’ve seen people doing it) – an Illegal argument exception is a much better one. Most of the time the underlying platform is the one throwing Null pointer exceptions.

It’s just a convention, but Null pointer exceptions should be extinguished when sighted.

So…

How can we handle Null

Take 1

The most straightforward and wide spread solution is to tell everyone to check the values for Null references. But that is a blacklist approach and, for this kind of problem, it doesn’t work. People forget stuff – and sooner or later someone will forget to do it and your 24/7 program will crash at 3AM on Sunday. Unit testing can help a lot here, but it requires work and you still cannot be 100% sure.

Can we do better?

Take 2

I would like to point out that the following method makes more sense in a statically typed language, where you have a compiler, checking your types for validity.

Most of the time we return Null as a value from a method that we want to return nothing, but some return value is expected and we throw Null at it.

Let’s have a look at the following Java code:

public String tld(String domain) {
    int dot = domain.lastIndexOf('.');
    if (dot == -1 || dot == domain.length()) {
        return null;
    } else {
        return domain.substring(dot + 1);
    }
}

The method above returns the top level domain part, or null if none found.
The “right” user code of this method is like this:

String d = tld("example.org");
if (d != null) {
    // Save to database
} else {
    // Show the user a message
}

But, as I pointed out earlier – there will be time that you will forget to make the check and the user will see a nasty exception, leaving him clueless about what she did wrong.

What we can do instead is signify our intentions that we might not return a value.
In Scala, we have the Option class – heavily inspired by Haskell‘s Maybe monad.
The Option class has two (final) subclasses²: Some and None. You return Some when you have something to return and None when you don’t.
Let’s look at the tld method implemented in Scala using the Option class:

def tld(domain: String): Option[String] = {
  val dot = domain lastIndexOf '.'
  if (dot == -1 || dot == domain.length) None
  else Some(domain.substring(dot + 1))
}

And the usage if it:

val com: String = tld("example.com").getOrElse("unknown")

That line above will return either the tld part the domain or "unknown" if none is found. Concise, isn’t it?
If we omit the getOrElse part the compiler with complain the we are not giving it a String value, but None instead.

// Does not compile!
val com: String = tld("example.com")

That way we are able to catch the error early (say 17:00 o’clock on Friday which is much better than 3:00 in the morning on Sunday).

What we’ve done? We told the type system that we might return a value, but not necessary. Specifying the return type of tld to be Option[String] instead of just String does just that.

So… Can we do that in Java?

Let’s give it a try.
We’ll have a JOption interface with JSome and JNone implementing classes:

public interface JOption<T> {
    public boolean isEmpty();
    public T getOrElse(T other);
}

The JNone class will mark the lack of value.

public class JNone<T> implements JOption<T> {
    public JNone() {}
 
    public boolean isEmpty() {
        return true;
    }
 
    public T getOrElse(T other) {
        return other;
    }
}

and the JSome class – the presence of it.

public class JSome<T> implements JOption<T> {
    private T value;
 
    public JSome(T value) {
        this.value = value;
    }
 
    public boolean isEmpty() {
        return false;
    }
 
    public T getOrElse(T other) {
        return value;
    }
}

And now our revised tld method, using the new JOption machinery:

public JOption<String> tld(String domain) {
    int dot = domain.lastIndexOf('.');
    if (dot == -1 || dot == domain.length()) {
        return new JNone<String>();
    } else {
        return new JSome<String>(domain.substring(dot + 1));
    }
}

And using it becomes:

String com = tld("example").getOrElse("unknown");

The tld method isn’t longer than our original Null-returning version, but now the compiler will help us, because if we write:

// Does not even compile
String com = tld("example");

we’ll get a compiler error.

Final thoughts

The proposed method probably isn’t the best and can be improved³, but it illustrates the concept.

Also, while this solution might help, the Null value still exists and people canwill (ab)use it.
Scala has a Null value as well – for compatibility with Java.

On the bright side – Haskell, Erlang and other functional languages don’t have anything like Null in them and use some form of the described approach.

But on the dark side we have Javascript – there you have undefined, null and NaN that adds even more codding horror. F# is in similar state.

Please, share your thoughts bellow.
__________________
¹ On 25 August 2009 InfoQ published the video.
² To be precise, None is an object (a singleton in Scala).
³ We can also add other methods to JOption like get() that will throw an exception in the case of JNone and return the value for JSome, as well as a proper equals implementation.

Megacorp138 HQ

Time goes by, John is now 28 and he is a leader of a small team, working on a very interesting project, which they have just started. From scratch! Life could not be better…

Unfortunately, soon things start to become worse. The time passes by and the project is delayed many times. Eventually it goes into production, but the users are finding new and strange bugs every day. In time all of the developers get frustrated and quit. John is now The Man of The Project. He knows (almost) everything about it and is a master of The Language.

Now John has kids, a house, a couple of cars, and even a better paid job. The project he was in charge of is now cleared of bugs, he knows most of the pieces in it, but from time to time there are some surprises that he does not expect.

John is 39. A new start-up comes along claiming that they have better product than the one John has developed.
Aah, we can crush them! They have nothing better than us — that was the mood in the company.

Few years fly by. The start-up is still doing well. The mood in MegaCorp138 has slightly changed to: We are much bigger and stronger than them. They have some extra features we don’t have, but we’ll beat them. John is the Head of the programming department now.

After few more years and some mood changes, MegaCorp138 is near to bankruptcy. The owners of the company decide to sell it to that small start-up they thought they will crush. John gets laid off.

John is 51 now, looking for a new job in the programming world. But he is having a very hard time in finding a new job, and, most importantly – he has no clue why? All these people are asking for completely different skills. John has been programming in The Language. He knows every single part of it. Of course, while he was working in MegaCorp138, now and then, a young chap would come and told him:

xUnit Test Patterns

– You should learn at least one new language per year.
– Why should I learn a new language? I’m using The Language!

Another chap tried to convince him to write tests.

Tests? I know The Language and The Product from inside and out. I don’t need tests. I’ve tested it and – here – it works. It’s on my screen. I don’t have the time to write 50% more code. I’m way too busy.

Those were commonplace pieces of advice to John.

Here is John today, still looking for a job. At one of the interviews the interviewer tells him a story.

A man was walking through the forest when he noticed a woodcutter. The man went closer and saw that the saw of the woodcutter was very blunt.
– Hey, your saw is very blunt. Why don’t you stop and sharpen it?
– STOP and sharpen it? — the woodcutter responded rudely — Can’t you see that I have a lot of work to do?

What a stupid woodcutter!, thought John.

Are you That woodcutter? Or That John?

___________
Many thanks to Nas OOD for reviewing this post.

Stefan’s talk about Ruby was on the target as usual. He is a very talented presenter and to be perfectly fair, I’ve seen him presenting before and borrowed (stolen) some ideas from him :), but mostly I got ideas from his source – the Ruby community – they are probably the best community when it comes to marketing and entertainment (and many other things as well).

The third part was a discussion about “The perfect programming language”. To be perfectly honest I didn’t understand some parts of it, but that was probably due to the fact that I was very tired.
Basically, what I could conclude is that Lisp seems to be an excellent language, with (way (too (many) (parenthesis))). :)

Here are the most important links from my presentation:

• Growing a langauge, by Guy Steele – an incredible talk by Guy Steele about programming in general!
• The Scala Programming Langauge
• The busy Java developer’s guide to Scala
• O’Reilly “Programming Scala” – an early preview of a book about Scala from O’Reilly.
• The Feel of Scala – A great talk on Scala by one of the core developers.

I would be happy to answer to any questions you have.

Below are the slides of the presentation. Basically only the code examples. I’ve removed the images and the jokes as they make little sense without context:

Take care.

Here are we now, 2009, the Java language does have generics, which seem not thought well enough¹, but doesn’t have operator overloading. It may never have it.
And here are we again, 2009, we have the Scala Programming Language. It can satisfy both requirements Guy mentions explicitly – operator overloading and generics. But Scala has pushed these concepts further.
Actually, it doesn’t have operator overloading – it has a way to do operator overloading by allowing people to define methods that look like operators.
Compare the use of Java’s BigInteger class

import java.math.BigInteger
def factorial(x: BigInteger): BigInteger =
  if (x == BigInteger.ZERO)
    BigInteger.ONE
  else
    x.multiply(factorial(x.subtract(BigInteger.ONE)))

and Scala’s wrapped version – BigInt

def factorial(x: BigInt): BigInt =
  if (x == 0) 1 else x * factorial(x - 1)

It looks natural, clean and concise².

Scala also has generics, but, yet again, pushed one idea further. They are called type parameters. Just like a method may receive objects as parameters, the same way it may receive types as parameters. This allows one to write methods (or classes) in a way that they work with any or some types without explicitly defining behavior for each supported type and puts the control in the hands of the developer that writes the class.

object Queue {
  // constructs a queue with initial elements ‘xs’
  def apply[T](xs: T*) = new Queue[T](xs.toList, Nil)
}

Here both the class Queue and the method apply are parametrized by type [T]².

Growing your language is not a new idea – in 1993, Paul Graham wrote about Bottom-up programming where he proudly stands behind this idea that a language should grow with the user and be grown by the user. Lisp have had this for years decades, but for (some (strange (reason))) never hit mainstream³.

A couple of years ago all this fuzz about Ruby and Domain Specific Languages, Internal Domain Specific Languages to be precise, hit the scene. And Ruby has a lot of ideas from Lisp in it, but with better syntax.

Ultimately, the perfect language will be one with no primitives – everything can be defined as a library and the language will be able to be grown by it’s users, but not one that is ultimately flexible as it will produce very different and divergent styles with hard-to-cache errors. In my opinion Scala is the one that stand the best chance of satisfying those requirements.

__________________
¹ Try using wildcards in a combination of framework and user code, where you do not own both.
² Examples are provided for completeness and not explained as their explanation will blur the main topic of this post.
³ Lisp being mainstream or not is arguable, without defining “mainstream”, but even it’s most passionate defenders will agree that in terms of popularity in can do a lot better.