Writing Your Own Storage Engine for Memcached

I am working full time on membase, which utilize the “engine interface” we're adding to Memcached. Being the one who designed the API and wrote the documentation, I can say that we do need more (and better) documentation without insulting anyone. This blog entry will be the first entry in mini-tutorial on how to write your own storage engine. I will try to cover all aspects of the engine interface while we're building an engine that stores all of the keys on files on the server.

This entry will cover the basic steps of setting up your development environment and cover the lifecycle of the engine.

Set up the development environment

The easiest way to get “up'n'running” is to install my development branch of the engine interface. Just execute the following commands:

Lets verify that the server works by executing the following commands:

Creating the filesystem engine

You might want to use autoconf to build your engine, but setting up autoconf is way beyond the scope of this tutorial. Let's just use the following Makefile instead.

I am doing most of my development on Solaris using the Sun Studio compilers, but I have added a section with settings for gcc there if you're using gcc. Just comment out lines for CC, CFLAGS and LDFLAGS and remove the # for the gcc alternatives.

In order for memcached to utilize your storage engine it needs to first load your module, and then create an instance the engine. You use the -E option to memcached to specify the name of the module memcached should load. With the module loaded memcached will look for a symbol named create_instance in the module to create an handle memcached can use to communicate with the engine. This is the first function we need to create, and it should have the following signature:

The purpose of this function is to provide the server a handle to our module, but we should not perform any kind of initialization of our engine yet. The reason for that is because the memcached server may not support the version of the API we provide. The intention is that the server should notify the engine with the “highest” interface version it supports through interface, and the engine must return a descriptor to one of those interfaces through the handle. If the engine don't support any of those interfaces it should return ENGINE_ENOTSUP.

So let's go ahead and define a engine descriptor for our example engine and create an implementation for create_instance:

If the interface we provide in create_instance is dropped from the supported interfaces in memcached, the core will call destroy() immediately. The memcached core guarantees that it will never use any pointers returned from the engine when destroy() is called.

So let's go ahead and implement our destroy() function. If you look at our implementation of create_instance you will see that we mapped destroy() to a function named fs_destroy():

If the core implements the interface we specify, the core will call a the initialize() method. This is the time where you should do all sort of initialization in your engine (like connecting to a database, initializing mutexes etc). The initialize function is called only once per instance returned from create_instance (even if the memcached core use multiple threads). The core will not call any other functions in the api before the initialization method returns.

We don't need any kind of initialization at this moment, so we can use the following initialization code:

If the engine returns anything else than ENGINE_SUCCESS, the memcached core will refuse to use the engine and call destroy()

In the next blog entry we will start adding functionality so that we can load our engine and handle commands from the client.