Mon Jan 8 23:56:42 ART 2007 Alberto Bertogli tagged 0.11 Mon Jan 8 23:54:29 ART 2007 Alberto Bertogli * Add a reference to the user guide in the README file. Mon Jan 8 23:50:30 ART 2007 Alberto Bertogli * Add the user guide. Mon Jan 8 19:15:14 ART 2007 Alberto Bertogli * Reduce tests memory footprint. This applies the same malloc() reduction we did in the last patch (20070108051156-57897-6f70641b48c356b12cadbb6cb84e82d615c860e2.gz) to the tests. It doesn't make any difference in performance, but it's makes a better example. Mon Jan 8 18:00:23 ART 2007 Alberto Bertogli * Fix the contains? test in the Python module. Mon Jan 8 02:11:56 ART 2007 Alberto Bertogli * Reduce do_get() malloc() size. Instead of doing malloc(128k) do malloc(70k) which is more appropriate given the max size of a TIPC packet (66000 bytes). This speed get operations up considerably, for instance for "./test1c 20000 k v" and "./test2c 20000 4 4": malloc(128k) malloc(70k) test1c 536426 376679 test2c 520247 381682 Now performance fits better with the expectations from the code analysis. Sun Jan 7 03:27:31 ART 2007 Alberto Bertogli * Update the TODO file. Sun Jan 7 03:25:05 ART 2007 Alberto Bertogli * Bring the python module up to date. This patch updates the python bindings to support multiple servers; allow the use of python object both as keys and as values, as long as they can be hashed and pickled (respectively); and document everything a bit. Sun Jan 7 02:46:21 ART 2007 Alberto Bertogli * Save vsize when replacing a cache entry. When setting some value that already is in the cache, we replace the value buffer but forgot to replace the value length. This patch fixes that by doing the obvious thing: replacing the old length with the new one. Sat Jan 6 15:16:16 ART 2007 Alberto Bertogli * Add two tests for multiple server access. This patch adds two tests that use multiple servers. They're the same tests as test2[dc], but use server ports 10, 11, 12 and 13. Sat Jan 6 15:14:34 ART 2007 Alberto Bertogli * Make the value buffer independant in test1* Sat Jan 6 13:34:55 ART 2007 Alberto Bertogli * Add nmdb_add_server() to libnmbd's manpage. Sat Jan 6 13:31:36 ART 2007 Alberto Bertogli * Implement multiple server support in the C library. This patch adds support for multiple servers in the C library by adding a nmbd_add_server() function, and adding a way of selecting the server to talk to based on the key of the request. It breaks the ABI. Mon Jan 1 23:22:55 ART 2007 Alberto Bertogli * Prepare to support multiple servers. This patch creates the infraestructure changes to support multiple servers, but doesn't make any significant change yet, just the function prototypes and the API/ABI. It breaks the ABI. Sun Nov 5 23:19:51 ART 2006 Alberto Bertogli * Fix a word in the BOLA license. I used an old version of the license that contained a typo: instead of "renewable" it said "renovable", which is not proper english. Fri Sep 15 02:46:10 ART 2006 Alberto Bertogli tagged 0.10 diff -rN -u old-nmdb/doc/guide.rst new-nmdb/doc/guide.rst --- old-nmdb/doc/guide.rst 1969-12-31 21:00:00.000000000 -0300 +++ new-nmdb/doc/guide.rst 2007-01-08 23:58:05.000000000 -0300 @@ -0,0 +1,408 @@ + +================ +nmdb User Guide +================ +:Author: Alberto Bertogli (albertito@gmail.com) + + +Introduction +============ + +nmdb_ is a simple and fast cache and database for TIPC_ clusters. It allows +applications in the cluster to use a centralized, shared cache and database in +a very easy way. It stores *(key, value)* pairs, with each key having only one +associated value. + +This document explains how to setup nmdb and a simple guide to writing +clients. It also includes a "quick start" section for the anxious. + + +Installing nmdb +=============== + +If you installed nmdb using your Linux distribution package system, you can +skip this section entirely. + + +Prerequisites +------------- + +Before you install nmdb, you will need the following software: + +- libevent_, a library for fast event handling. +- `Linux kernel`_ 2.6.16 or newer, compiled with TIPC_ support. +- QDBM_, for the database backend. + + +Compiling and installing +------------------------ + +There are three components of the nmdb tarball: the server in the *nmdb/* +directory, the C library in *libnmdb/*, and the Python module in *python/*. + +- To install the server, run ``cd nmdb; make install``. +- To install the C library, run ``cd libnmdb; make install; ldconfig``. +- To install the Python module, run ``cd python; python setup.py install``. + + +Quick start +=========== + +For a very quick start, using a single host, you can do the following:: + + # dpmgr create /tmp/nmdb-db # create the backend database + # nmdb -d /tmp/nmdb-db # start the server + +At this point you have created a database and started the server. An easy and +simple way to test it is to use the python module, like this:: + + # python + Python 2.5 (r25:51908, Sep 21 2006, 20:38:23) + [GCC 4.1.1 (Gentoo 4.1.1)] on linux2 + Type "help", "copyright", "credits" or "license" for more information. + >>> import nmdb # import the module + >>> db = nmdb.DB() # connect to the server + >>> db['x'] = 1 # store some data + >>> db[(1, 2)] = (2, 6) + >>> print db['x'], db[(1, 2)] # retreive the values + 1 (3, 5) + >>> del db['x'] # delete from the database + +Everything should have worked as shown, and you are now ready to use some +nmdb application, or develop your own. + +If you want to use this with several machines, read the next section to find +out how to setup a simple TIPC cluster. + + +TIPC setup +========== + +If you want to use the server and the clients in different machines, you need +to setup your TIPC network. If you just want to run everything in one machine, +or you already have a TIPC network set up, you can skip this section. + +Before we begin, all the machines should already be connected in an Ethernet +LAN, and have the tipc-config application that should come with your Linux +distribution with a package named "tipcutils" or similar (if it doesn't, you +can find it at http://tipc.sourceforge.net/download.html). + +The only thing you will need to do is assign each machine a TIPC address and +specify which interface to use for the network connection. You do it like +this:: + + # tipc-config -a=1.1.10 -be=eth:eth0 + +The *-a* parameter specifies the address, and *-be* the type and name of the +interface to use. + +Addresses are composed of three integers. They represent the zone number, the +cluster number, and the node number respectively. The zone number and cluster +number should be the same for all nodes in your network, so you should change +the last one for each machine. Each machine can have only one address. + +That should be enough to get you started for a small network. If you have a +very big network, or want to use some of the advanced TIPC features like link +redundancy, you should read TIPC's docs. + + +Example +------- + +If you have five machines, you can assign each one their address like this:: + + box1# tipc-config -a=1.1.1 -be=eth:eth0 + box2# tipc-config -a=1.1.2 -be=eth:eth0 + box3# tipc-config -a=1.1.3 -be=eth:eth0 + box4# tipc-config -a=1.1.4 -be=eth:eth0 + box5# tipc-config -a=1.1.5 -be=eth:eth0 + + +Starting the server +=================== + +Before starting the server, there are some things you need to know about it: + +Port numbers + Each server instance in your network (even the ones running in the same + machine) should get a **unique** port to listen to requests. Ports identify + an application instance inside the whole network, not just the machine as in + TCP/IP. + + The port space is very very large, and it's private to nmdb, so you can + choose numbers without fear of colliding with other TIPC applications. The + default port is 10. + + So, if you are going to start more than one nmdb server, **be careful**. If + you assign two active servers the same port you will get no error, but + everything will act weird. + +Cache size + nmdb's cache is a main component of the server. In fact you can use it + exclusively for caching purposes, like memcached_. So the size becomes an + important issue if you have performance requirements. + + It is only possible to limit the cache size by the maximum number of objects + in the cache. + +Backend database + You will need to create a backend database using QDBM_'s utilities. This is + quite simple, just run ``dpmgr create /path/to/the/database`` and you're + done. + + If for some reason (hardware failure, for instance) the database becomes + corrupt, you should use QDBM's utilities to fix it. It shouldn't happen, so + it's a good idea to report it if it does. + + QDBM databases are not meant to be shared among processes, so avoid having + other processes using them. + +Database redundancy + If you want to have redundancy over the database, you can start a "passive + server" along a normal one using the same port number. It will listen to + database requests and act upon them, but it will not reply anything. + + It is only useful to keep a live mirror of the database. Note that it does + not do replication or failure detection, it's just a mirror. + + This is the only case where you want to start two servers with the same port. + +Distributed queries + If you have more than one server in the network, the library can distribute + the queries among them. This is entirely done on the client side and the + server doesn't know about it. + + +Now that you know all that, starting a server should be quite simple: first +create the database as explained above, and then run the daemon with +``nmdb -d /path/to/the/database``. + +To change the port, use ``-l port``, to change the cache size, use ``-c nobj`` +(where *nobj* is the number of objects in thousands), to make the server +passive, use ``-p``. Of course you won't remember all that (I know I don't), +that's why ``-h`` is your friend. + +Nothing prevents you from starting more than one server in the same machine, +so be careful to select different ports and databases for each one. + + +Example +------- + +Following the previous example, if you want to start three servers you can do +it like this:: + + box1# ndbm -d /var/lib/nmdb/db-1 -l 11 + box2# ndbm -d /var/lib/nmdb/db-2 -l 12 + box3# ndbm -d /var/lib/nmdb/db-3 -l 13 + + +Writing clients +=============== + +At the moment you can write clients in C (documented in the *libnmdb*'s +manpage) and in Python (documented using Python docstrings). In this guide we +will give some examples of common use as an introduction, you should consult +the appropriate documentation when doing serious development. + +Before we begin, you should know about the following things: + +Thread safety + While the library itself is thread safe, neither the C library connections + nor the Python objects are. So don't share *nmdb_t* variables (C) or + *nmdb.** objects (Python) among threads; instead, create one for each thread + that needs it. + +Available operations + You can request the server to do three operations: *set* a value to a key, + *get* the value associated with the given key, and *delete* a given key + (with its associated value). + +Request modes + For each operation, you will have three different modes available: + + - A *normal mode*, which makes the operation impact on the database + asynchronously (ie. the functions return right after the operation was + queued, there is no completion notification). + - A *synchronous mode* similar to the previous one, but when the functions + return, the operation has hit the disk. + - A *cache-only mode* where the operations do not impact the database, only + the cache, and can be used to implement distributed caching in a similar + way to memcached_. + + Be careful with the last one, because mixing cache-only with database + operations is a recipe for disaster. + +Atomicity and coherence + All operations are atomic, and synchronous and asynchronous operations are + fully coherent. + +Distributed queries + You can distribute your queries among several servers, and this is entirely + done on the client side. To do this, you should add each server (identified + by their port numbers) to the connection **before beginning to interact with + them**. + + +For all examples we will assume that you have three servers running in your +network, in ports 11, 12 and 13. + + +The Python module +------------------ + +The Python module it's quite easy to use, because its interface is very +similar to a dictionary. It has similar limitations regarding the key (it must +be an object you can use as a key in a dictionary), and the values must be +pickable objects (see the *pickle* module documentation for more information). +In short, you should only use number, strings or tuples as keys, and simple +objects as values, unless you know what you are doing. + +To start a connection to the servers, you must first decide which mode you are +going to use: the normal database-backed mode, database-backed with +synchronous access, or cache only. Let's say you want to use the normal mode +and connect to the server at port 11, and then add the other two servers:: + + import nmdb + db = nmdb.DB(11) + db.add_server(12) + db.add_server(13) + +Now you're ready to use it. Let's suppose you want to write a recursive +function to calculate the factorial of a number. But before doing the +calculation, you can check if the previous factorial already is in the +database to avoid recalculating it:: + + def fact(n): + if n == 1: + return 1 + if db.has_key(n): + return db[n] + + result = n * fact(n - 1) + db[n] = result + return result + +That was easy, wasn't it? You can use the same trick for SQL queries, complex +distributed calculations, geographical data processing, whatever you want. + +Now let's have some fun and do something a little advanced: a decorator for a +distributed function cache. If Python magic scares you, look away and skip to +the next section. + +Some functions (usually the mathematical ones) have the property that the +value they return depends only on the parameters, and not on the context. So +they can be cached, using the parameters as keys, with the function's result +as their associated values. Applying this technique is commonly known as +*memoization*, and when we apply it to a function we say we're *memoizing* it. + +We can use a local dictionary to cache the data, but that would mean we would +have to write some cache management code to avoid using too much memory, and, +worse of all, each instance of the code running in the network would have its +own private cache and can't reuse calculations performed by other instances. +Instead, we can use nmdb to make a cache that is shared among the network. + +The functions are usually restricted to using simple types as input, like +numbers, strings, tuples or dictionaries. We will take advantage of this by +using as a key to the cache the string ``--``. So to cache an invocation +like ``mod.f(1, (2, 6))`` that returns ``26``, we want to have the following +association in the database: ``mod-f-(1, (2, 6)) = 26``. + +We will use nmdb in cache-only mode, where the things we store are not saved +permanently to a database, but live in the server's memory. This is very +similar to what we did before, and has the advantage of not having to write +our own cache management routines:: + + import nmdb + db = nmdb.Cache(11) + db.add_server(12) + db.add_server(13) + +Let's write the decorator:: + + def shared_memoize(f): + def newf(*args, **kwargs): + key = '%s-%s-%s-%s' % (f.__module__, f.__name__, + repr(args), repr(kwargs)) + if key in db: + return db[key] + r = f(*args, **kwargs) + db[key] = r + return r + return newf + +Now we can use it with a normal implementation of the recursive factorial +function like we did before, and a function that calculates tetrations_:: + + @shared_memoize + def fact(n): + if n == 1: + return 1 + return n * fact(n - 1) + + @shared_memoize + def tetration(a, b): + if b == 1: + return a + return pow(a, tetration(a, b - 1)) + +As you can see, the module is very easy to use, but you can do useful things +with it. For more information you can read the module's built-in +documentation. + + +The C library +------------- + +The C library is in essence similar to the Python module, so we won't make a +very long example here, only a brief display of the available functions. + +Let's begin by creating a "nmdb descriptor" which is of type *nmdb_t*, and +connecting it to your three servers:: + + unsigned char *key, *val; + size_t ksize, vsize; + nmdb_t *db; + + db = nmdb_connect(11); + nmdb_add_server(db, 12); + nmdb_add_server(db, 13); + +Now you can do some operations (allocations and checks are not shown for brevity):: + + r = nmdb_set(db, key, ksize, val, vsize); + ... + r = nmdb_get(db, key, ksize, val, vsize); + ... + r = nmdb_del(db, key, ksize); + +And finally close and free the connection:: + + nmdb_free(db); + +The operation functions have variants for cache-only (*nmdb_cache_**) and synchronous +operation (*nmdb_sync_**). For more information you should check the manpage. + + +Where to go from here +===================== + +The best place to go from here is to your text editor, to start writing some +simple clients to play with. + +If you are in doubt about something, you can consult the manpages or the +documentation inside the *doc/* directory. And if you can't find an answer to +your question there, you can ask me, Alberto Bertogli, at +*albertito@gmail.com*. + + + +.. _nmdb: http://auriga.wearlab.de/~alb/nmdb/ +.. _libevent: http://www.monkey.org/~provos/libevent/ +.. _TIPC: http://tipc.sf.net +.. _memcached: http://www.danga.com/memcached/ +.. _QDBM: http://qdbm.sf.net +.. _`Linux kernel`: http://kernel.org +.. _tetrations: http://en.wikipedia.org/wiki/Tetration + diff -rN -u old-nmdb/libnmdb/libnmdb.3 new-nmdb/libnmdb/libnmdb.3 --- old-nmdb/libnmdb/libnmdb.3 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/libnmdb/libnmdb.3 2007-01-08 23:58:05.000000000 -0300 @@ -6,7 +6,7 @@ .B #include .sp .BI "nmdb_t *nmdb_init(int " port ");" -.sp +.BI "int nmdb_add_server(nmdb_t *" db ", int " port ");" .BI "int nmdb_free(nmdb_t *" db ");" .sp .BI "int nmdb_set(nmdb_t *" db "," @@ -47,6 +47,12 @@ is the TIPC port to use. If you are in doubt, use -1 which will use the default value. +Optionally, you can add more servers to the server pool, using +.BR nmdb_add_server (). +You can add any number of servers, and each time a request is made, one will +be selected. Be aware that you should add all the servers before start using +the database. + To dispose a connection, use .BR nmdb_free (). diff -rN -u old-nmdb/libnmdb/libnmdb.c new-nmdb/libnmdb/libnmdb.c --- old-nmdb/libnmdb/libnmdb.c 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/libnmdb/libnmdb.c 2007-01-08 23:58:05.000000000 -0300 @@ -18,71 +18,145 @@ #define ID_CODE 1 +/* Create a nmdb_t and set the first server to port. If port is < 0, the + * standard port is used. */ nmdb_t *nmdb_init(int port) { int fd; nmdb_t *db; + struct nmdb_srv *server; db = malloc(sizeof(nmdb_t)); if (db == NULL) { return NULL; } - if (port < 0) - port = SERVER_INST; + server = malloc(sizeof(struct nmdb_srv)); + if (server == NULL) { + free(db); + return NULL; + } - db->srvsa.family = AF_TIPC; - db->srvsa.addrtype = TIPC_ADDR_NAMESEQ; - db->srvsa.addr.nameseq.type = SERVER_TYPE; - db->srvsa.addr.nameseq.lower = port; - db->srvsa.addr.nameseq.upper = port; - db->srvsa.scope = TIPC_CLUSTER_SCOPE; - db->srvlen = (socklen_t) sizeof(db->srvsa); + db->servers = server; + db->nservers = 1; + /* the fd is shared among the different servers, because we use + * sendto() directly instead of connecting sockets */ fd = socket(AF_TIPC, SOCK_RDM, 0); if (fd < 0) { + free(db->servers); free(db); return NULL; } db->fd = fd; + if (port < 0) + port = SERVER_INST; + + server->port = port; + server->srvsa.family = AF_TIPC; + server->srvsa.addrtype = TIPC_ADDR_NAMESEQ; + server->srvsa.addr.nameseq.type = SERVER_TYPE; + server->srvsa.addr.nameseq.lower = port; + server->srvsa.addr.nameseq.upper = port; + server->srvsa.scope = TIPC_CLUSTER_SCOPE; + server->srvlen = (socklen_t) sizeof(server->srvsa); + return db; } +/* Compare two servers, using their ports. It is used internally to keep the + * server array sorted with qsort(). */ +static int compare_servers(const void *s1, const void *s2) +{ + struct nmdb_srv *srv1 = (struct nmdb_srv *) s1; + struct nmdb_srv *srv2 = (struct nmdb_srv *) s2; + + if (srv1->port < srv2->port) + return -1; + else if (srv1->port == srv2->port) + return 0; + else + return 1; +} + +/* Add a server to the db connection. Requests will select which server to use + * by hashing the key. */ +int nmdb_add_server(nmdb_t *db, int port) +{ + struct nmdb_srv *newsrv, *newarray; + + newarray = realloc(db->servers, + sizeof(struct nmdb_srv) * (db->nservers + 1)); + if (newarray == NULL) { + return 0; + } + db->servers = newarray; + db->nservers++; + + newsrv = &(db->servers[db->nservers - 1]); + + if (port < 0) + port = SERVER_INST; + + newsrv->port = port; + newsrv->srvsa.family = AF_TIPC; + newsrv->srvsa.addrtype = TIPC_ADDR_NAMESEQ; + newsrv->srvsa.addr.nameseq.type = SERVER_TYPE; + newsrv->srvsa.addr.nameseq.lower = port; + newsrv->srvsa.addr.nameseq.upper = port; + newsrv->srvsa.scope = TIPC_CLUSTER_SCOPE; + newsrv->srvlen = (socklen_t) sizeof(newsrv->srvsa); + + /* keep the list sorted by port, so we can do a reliable selection */ + qsort(db->servers, db->nservers, sizeof(struct nmdb_srv), + compare_servers); + + return 1; +} + +/* Frees a nmdb_t structure created with nmdb_init(). */ int nmdb_free(nmdb_t *db) { close(db->fd); + if (db->servers != NULL) + free(db->servers); free(db); return 1; } -static int srv_send(nmdb_t *db, const unsigned char *buf, size_t bsize) +/* Used internally to send a buffer to the given server. */ +static int srv_send(nmdb_t *db, struct nmdb_srv *srv, + const unsigned char *buf, size_t bsize) { ssize_t rv; - rv = sendto(db->fd, buf, bsize, 0, (struct sockaddr *) &(db->srvsa), - db->srvlen); + rv = sendto(db->fd, buf, bsize, 0, (struct sockaddr *) &(srv->srvsa), + srv->srvlen); if (rv <= 0) return 0; return 1; } -static ssize_t srv_recv(nmdb_t *db, unsigned char *buf, size_t bsize) +/* Used internally to receive a buffer. */ +static ssize_t srv_recv(nmdb_t *db, struct nmdb_srv *srv, + unsigned char *buf, size_t bsize) { ssize_t rv; rv = recv(db->fd, buf, bsize, 0); return rv; - } -static uint32_t get_rep(nmdb_t *db, unsigned char *buf, size_t bsize, +/* Used internally to get and parse replies from the server. */ +static uint32_t get_rep(nmdb_t *db, struct nmdb_srv *srv, + unsigned char *buf, size_t bsize, unsigned char **payload, size_t *psize) { ssize_t t; uint32_t id, reply; - t = srv_recv(db, buf, bsize); + t = srv_recv(db, srv, buf, bsize); if (t < 4 + 4) { return -1; } @@ -103,15 +177,45 @@ return reply; } +/* Hash function used internally by select_srv(). See RFC 1071. */ +uint32_t checksum(const unsigned char *buf, size_t bsize) +{ + uint32_t sum = 0; + + while (bsize > 1) { + sum += * (uint16_t *) buf++; + bsize -= 2; + } + + if (bsize > 0) + sum += * (uint8_t *) buf; + + while (sum >> 16) + sum = (sum & 0xffff) + (sum >> 16); + + return ~sum; +} + +/* Used internally to select which server to use for the given key. */ +static struct nmdb_srv *select_srv(nmdb_t *db, + const unsigned char *key, size_t ksize) +{ + uint32_t n; + + n = checksum(key, ksize) % db->nservers; + return &(db->servers[n]); +} -static ssize_t do_get(nmdb_t *db, const unsigned char *key, size_t ksize, +static ssize_t do_get(nmdb_t *db, + const unsigned char *key, size_t ksize, unsigned char *val, size_t vsize, int impact_db) { ssize_t rv, t; unsigned char *buf, *p; size_t bsize, reqsize, psize; uint32_t request, reply; + struct nmdb_srv *srv; if (impact_db) { request = REQ_GET; @@ -126,9 +230,9 @@ * vsize bytes key. * * We don't know vsize beforehand, but we do know TIPC's max packet is - * 66000. We malloc 128k just in case. + * 66000. We malloc 70k just in case. */ - bsize = 128 * 1024; + bsize = 70 * 1024; buf = malloc(bsize); if (buf == NULL) return -1; @@ -140,13 +244,14 @@ memcpy(p, key, ksize); reqsize = 3 * 4 + ksize; - t = srv_send(db, buf, reqsize); + srv = select_srv(db, key, ksize); + t = srv_send(db, srv, buf, reqsize); if (t <= 0) { rv = -1; goto exit; } - reply = get_rep(db, buf, bsize, &p, &psize); + reply = get_rep(db, srv, buf, bsize, &p, &psize); if (reply == REP_CACHE_MISS || reply == REP_NOTIN) { rv = 0; @@ -196,6 +301,7 @@ unsigned char *buf, *p; size_t bsize; uint32_t request, reply; + struct nmdb_srv *srv; if (impact_db) { if (async) @@ -226,13 +332,14 @@ p += ksize; memcpy(p, val, vsize); - t = srv_send(db, buf, bsize); + srv = select_srv(db, key, ksize); + t = srv_send(db, srv, buf, bsize); if (t <= 0) { rv = -1; goto exit; } - reply = get_rep(db, buf, bsize, NULL, NULL); + reply = get_rep(db, srv, buf, bsize, NULL, NULL); if (reply == REP_OK) { rv = 1; @@ -275,6 +382,7 @@ unsigned char *buf; size_t bsize; uint32_t request, reply; + struct nmdb_srv *srv; if (impact_db) { if (async) @@ -301,13 +409,14 @@ * ((uint32_t *) buf + 2) = htonl(ksize); memcpy(buf + 3 * 4, key, ksize); - t = srv_send(db, buf, bsize); + srv = select_srv(db, key, ksize); + t = srv_send(db, srv, buf, bsize); if (t <= 0) { rv = -1; goto exit; } - reply = get_rep(db, buf, bsize, NULL, NULL); + reply = get_rep(db, srv, buf, bsize, NULL, NULL); if (reply == REP_OK) { rv = 1; diff -rN -u old-nmdb/libnmdb/LICENSE new-nmdb/libnmdb/LICENSE --- old-nmdb/libnmdb/LICENSE 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/libnmdb/LICENSE 2007-01-08 23:58:05.000000000 -0300 @@ -24,7 +24,7 @@ 5. Help someone who needs it: sign up for some volunteer work or help your neighbour paint the house. 6. Don't waste. Anything, but specially energy that comes from natural - non-renovable resources. Extra points if you discover or invent something + non-renewable resources. Extra points if you discover or invent something to replace them. 7. Be tolerant. Everything that's good in nature comes from cooperation. diff -rN -u old-nmdb/libnmdb/Makefile new-nmdb/libnmdb/Makefile --- old-nmdb/libnmdb/Makefile 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/libnmdb/Makefile 2007-01-08 23:58:05.000000000 -0300 @@ -13,7 +13,7 @@ PREFIX=/usr/local -OBJS = libnmdb.o test1c.o test1d.o test2c.o test2d.o +OBJS = libnmdb.o test1c.o test1d.o test2c.o test2cm.o test2d.o test2dm.o default: all @@ -30,7 +30,7 @@ $(AR) cr libnmdb.a libnmdb.o -tests: test1c test1d test2c test2d +tests: test1c test1d test2c test2cm test2d test2dm test1c: test1c.o libnmdb.a $(CC) $(CFLAGS) test1c.o libnmdb.a -o test1c @@ -41,9 +41,15 @@ test2c: test2c.o libnmdb.a $(CC) $(CFLAGS) test2c.o libnmdb.a -o test2c +test2cm: test2cm.o libnmdb.a + $(CC) $(CFLAGS) test2cm.o libnmdb.a -o test2cm + test2d: test2d.o libnmdb.a $(CC) $(CFLAGS) test2d.o libnmdb.a -o test2d +test2dm: test2dm.o libnmdb.a + $(CC) $(CFLAGS) test2dm.o libnmdb.a -o test2dm + install: libs install -d $(PREFIX)/lib @@ -62,7 +68,8 @@ $(CC) $(CFLAGS) -c $< -o $@ clean: - rm -f $(OBJS) libnmdb.so libnmdb.a test1c test1d test2c test2d + rm -f $(OBJS) libnmdb.so libnmdb.a + rm -f test1c test1d test2c test2cm test2d test2dm rm -f *.bb *.bbg *.da *.gcov gmon.out .PHONY: default all libs tests install clean diff -rN -u old-nmdb/libnmdb/nmdb.h new-nmdb/libnmdb/nmdb.h --- old-nmdb/libnmdb/nmdb.h 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/libnmdb/nmdb.h 2007-01-08 23:58:05.000000000 -0300 @@ -6,13 +6,20 @@ #include /* socklen_t */ #include /* struct sockaddr_tipc */ -typedef struct nmdb_t { - int fd; +struct nmdb_srv { + int port; struct sockaddr_tipc srvsa; socklen_t srvlen; +}; + +typedef struct nmdb_t { + int fd; + unsigned int nservers; + struct nmdb_srv *servers; } nmdb_t; nmdb_t *nmdb_init(int port); +int nmdb_add_server(nmdb_t *db, int port); int nmdb_free(nmdb_t *db); ssize_t nmdb_get(nmdb_t *db, const unsigned char *key, size_t ksize, diff -rN -u old-nmdb/libnmdb/test1c.c new-nmdb/libnmdb/test1c.c --- old-nmdb/libnmdb/test1c.c 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/libnmdb/test1c.c 2007-01-08 23:58:05.000000000 -0300 @@ -12,7 +12,7 @@ int main(int argc, char **argv) { int i, r, times; - unsigned char *key, *val; + unsigned char *key, *val, *gval; size_t ksize, vsize; unsigned long elapsed, misses = 0; nmdb_t *db; @@ -46,11 +46,11 @@ elapsed = timer_stop(); printf("%lu\n", elapsed); - val = malloc(128 * 1024); + gval = malloc(70 * 1024); printf("get... "); timer_start(); for (i = 0; i < times; i++) { - r = nmdb_cache_get(db, key, ksize, val, vsize); + r = nmdb_cache_get(db, key, ksize, gval, vsize); if (r < 0) { perror("Get"); return 1; @@ -60,7 +60,7 @@ } elapsed = timer_stop(); printf("%lu\n", elapsed); - free(val); + free(gval); printf("get misses: %ld\n", misses); diff -rN -u old-nmdb/libnmdb/test1d.c new-nmdb/libnmdb/test1d.c --- old-nmdb/libnmdb/test1d.c 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/libnmdb/test1d.c 2007-01-08 23:58:05.000000000 -0300 @@ -12,7 +12,7 @@ int main(int argc, char **argv) { int i, r, times; - unsigned char *key, *val; + unsigned char *key, *val, *gval; size_t ksize, vsize; unsigned long elapsed, misses = 0; nmdb_t *db; @@ -46,11 +46,11 @@ elapsed = timer_stop(); printf("%lu\n", elapsed); - val = malloc(128 * 1024); + gval = malloc(70 * 1024); printf("get... "); timer_start(); for (i = 0; i < times; i++) { - r = nmdb_get(db, key, ksize, val, vsize); + r = nmdb_get(db, key, ksize, gval, vsize); if (r < 0) { perror("Get"); return 1; @@ -60,7 +60,7 @@ } elapsed = timer_stop(); printf("%lu\n", elapsed); - free(val); + free(gval); printf("get misses: %ld\n", misses); diff -rN -u old-nmdb/libnmdb/test2c.c new-nmdb/libnmdb/test2c.c --- old-nmdb/libnmdb/test2c.c 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/libnmdb/test2c.c 2007-01-08 23:58:05.000000000 -0300 @@ -64,7 +64,7 @@ memset(key, 0, ksize); free(val); - val = malloc(128 * 1024); + val = malloc(70 * 1024); timer_start(); for (i = 0; i < times; i++) { * (int *) key = i; diff -rN -u old-nmdb/libnmdb/test2cm.c new-nmdb/libnmdb/test2cm.c --- old-nmdb/libnmdb/test2cm.c 1969-12-31 21:00:00.000000000 -0300 +++ new-nmdb/libnmdb/test2cm.c 2007-01-08 23:58:05.000000000 -0300 @@ -0,0 +1,114 @@ + +#include +#include +#include +#include +#include + +#include "nmdb.h" +#include "timer.h" + + +int main(int argc, char **argv) +{ + int i, r, times; + unsigned char *key, *val, *gval; + size_t ksize, vsize; + unsigned long s_elapsed, g_elapsed, d_elapsed, misses = 0; + nmdb_t *db; + + if (argc != 4) { + printf("Usage: test2 TIMES KSIZE VSIZE\n"); + return 1; + } + + times = atoi(argv[1]); + ksize = atoi(argv[2]); + vsize = atoi(argv[3]); + if (times < 1) { + printf("Error: TIMES must be >= 1\n"); + return 1; + } + if (ksize < sizeof(int) || vsize < sizeof(int)) { + printf("Error: KSIZE and VSIZE must be >= sizeof(int)\n"); + return 1; + } + + key = malloc(ksize); + memset(key, 0, ksize); + val = malloc(vsize); + memset(val, 0, vsize); + + if (key == NULL || val == NULL) { + perror("Error: malloc()"); + return 1; + } + + db = nmdb_init(-1); + if (db == NULL) { + perror("nmdb_init() failed"); + return 1; + } + + if (!nmdb_add_server(db, 11) || + !nmdb_add_server(db, 12) || + !nmdb_add_server(db, 13)) { + perror("nmdb_add_server() failed"); + return 1; + } + + timer_start(); + for (i = 0; i < times; i++) { + * (int *) key = i; + * (int *) val = i; + r = nmdb_cache_set(db, key, ksize, val, vsize); + if (r < 0) { + perror("Set"); + return 1; + } + } + s_elapsed = timer_stop(); + + memset(key, 0, ksize); + gval = malloc(70 * 1024); + timer_start(); + for (i = 0; i < times; i++) { + * (int *) key = i; + r = nmdb_cache_get(db, key, ksize, gval, vsize); + if (r < 0) { + perror("Get"); + return 1; + } else if (r == 0) { + misses++; + continue; + } + * (int *) val = i; + if (memcmp((void *) val, (void *) gval, vsize) != 0) { + printf("Values differ for key %s: %s - %s\n", + key, val, gval); + printf("i: %d\n", i); + return 1; + } + } + g_elapsed = timer_stop(); + free(gval); + free(val); + + timer_start(); + for (i = 0; i < times; i++) { + * (int *) key = i; + r = nmdb_cache_del(db, key, ksize); + if (r < 0) { + perror("Del"); + return 1; + } + } + d_elapsed = timer_stop(); + printf("%lu %lu %lu %lu\n", s_elapsed, g_elapsed, d_elapsed, misses); + + free(key); + nmdb_free(db); + + return 0; +} + diff -rN -u old-nmdb/libnmdb/test2d.c new-nmdb/libnmdb/test2d.c --- old-nmdb/libnmdb/test2d.c 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/libnmdb/test2d.c 2007-01-08 23:58:05.000000000 -0300 @@ -64,7 +64,7 @@ memset(key, 0, ksize); free(val); - val = malloc(128 * 1024); + val = malloc(70 * 1024); timer_start(); for (i = 0; i < times; i++) { * (int *) key = i; diff -rN -u old-nmdb/libnmdb/test2dm.c new-nmdb/libnmdb/test2dm.c --- old-nmdb/libnmdb/test2dm.c 1969-12-31 21:00:00.000000000 -0300 +++ new-nmdb/libnmdb/test2dm.c 2007-01-08 23:58:05.000000000 -0300 @@ -0,0 +1,106 @@ + +#include +#include +#include +#include +#include + +#include "nmdb.h" +#include "timer.h" + + +int main(int argc, char **argv) +{ + int i, r, times; + unsigned char *key, *val; + size_t ksize, vsize; + unsigned long s_elapsed, g_elapsed, d_elapsed, misses = 0; + nmdb_t *db; + + if (argc != 4) { + printf("Usage: test2 TIMES KSIZE VSIZE\n"); + return 1; + } + + times = atoi(argv[1]); + ksize = atoi(argv[2]); + vsize = atoi(argv[3]); + if (times < 1) { + printf("Error: TIMES must be >= 1\n"); + return 1; + } + if (ksize < sizeof(int) || vsize < sizeof(int)) { + printf("Error: KSIZE and VSIZE must be >= sizeof(int)\n"); + return 1; + } + + key = malloc(ksize); + memset(key, 0, ksize); + val = malloc(vsize); + memset(val, 0, vsize); + + if (key == NULL || val == NULL) { + perror("Error: malloc()"); + return 1; + } + + db = nmdb_init(-1); + if (db == NULL) { + perror("nmdb_init() failed"); + return 1; + } + + if (!nmdb_add_server(db, 11) || + !nmdb_add_server(db, 12) || + !nmdb_add_server(db, 13)) { + perror("nmdb_add_server() failed"); + return 1; + } + + timer_start(); + for (i = 0; i < times; i++) { + * (int *) key = i; + * (int *) val = i; + r = nmdb_set(db, key, ksize, val, vsize); + if (r < 0) { + perror("Set"); + return 1; + } + } + s_elapsed = timer_stop(); + + memset(key, 0, ksize); + free(val); + val = malloc(70 * 1024); + timer_start(); + for (i = 0; i < times; i++) { + * (int *) key = i; + r = nmdb_get(db, key, ksize, val, vsize); + if (r < 0) { + perror("Get"); + return 1; + } else if (r == 0) { + misses++; + } + } + g_elapsed = timer_stop(); + free(val); + + timer_start(); + for (i = 0; i < times; i++) { + * (int *) key = i; + r = nmdb_del(db, key, ksize); + if (r < 0) { + perror("Del"); + return 1; + } + } + d_elapsed = timer_stop(); + printf("%lu %lu %lu %lu\n", s_elapsed, g_elapsed, d_elapsed, misses); + + free(key); + nmdb_free(db); + + return 0; +} + diff -rN -u old-nmdb/nmdb/cache.c new-nmdb/nmdb/cache.c --- old-nmdb/nmdb/cache.c 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/nmdb/cache.c 2007-01-08 23:58:05.000000000 -0300 @@ -242,6 +242,7 @@ } free(e->val); e->val = v; + e->vsize = vsize; memcpy(e->val, val, vsize); /* promote the entry to the top of the list if necessary */ diff -rN -u old-nmdb/python/LICENSE new-nmdb/python/LICENSE --- old-nmdb/python/LICENSE 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/python/LICENSE 2007-01-08 23:58:05.000000000 -0300 @@ -24,7 +24,7 @@ 5. Help someone who needs it: sign up for some volunteer work or help your neighbour paint the house. 6. Don't waste. Anything, but specially energy that comes from natural - non-renovable resources. Extra points if you discover or invent something + non-renewable resources. Extra points if you discover or invent something to replace them. 7. Be tolerant. Everything that's good in nature comes from cooperation. diff -rN -u old-nmdb/python/nmdb_ll.c new-nmdb/python/nmdb_ll.c --- old-nmdb/python/nmdb_ll.c 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/python/nmdb_ll.c 2007-01-08 23:58:05.000000000 -0300 @@ -35,6 +35,23 @@ } +/* add server */ +static PyObject *db_add_server(nmdbobject *db, PyObject *args) +{ + int port; + int rv; + + if (!PyArg_ParseTuple(args, "i:add_server", &port)) { + return NULL; + } + + Py_BEGIN_ALLOW_THREADS + rv = nmdb_add_server(db->db, port); + Py_END_ALLOW_THREADS + + return PyLong_FromLong(rv); +} + /* cache set */ static PyObject *db_cache_set(nmdbobject *db, PyObject *args) { @@ -222,6 +239,7 @@ /* nmdb method table */ static PyMethodDef nmdb_methods[] = { + { "add_server", (PyCFunction) db_add_server, METH_VARARGS, NULL }, { "cache_set", (PyCFunction) db_cache_set, METH_VARARGS, NULL }, { "cache_get", (PyCFunction) db_cache_get, METH_VARARGS, NULL }, { "cache_delete", (PyCFunction) db_cache_delete, METH_VARARGS, NULL }, diff -rN -u old-nmdb/python/nmdb.py new-nmdb/python/nmdb.py --- old-nmdb/python/nmdb.py 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/python/nmdb.py 2007-01-08 23:58:05.000000000 -0300 @@ -1,38 +1,98 @@ -# -# libnmdb python wrapper -# Alberto Bertogli (albertito@gmail.com) -# +""" +libnmdb python wrapper +This module is a wrapper for the libnmdb, the C library used to implement +clients to the nmdb server. + +It provides three similar classes: DB, SyncDB and Cache. They all present the +same dictionary-alike interface, but differ in how they interact with the +server. + +The DB class allows you to set, get and delete (key, value) pairs from the +database; the SyncDB class works like DB, but does so in a synchronous way; and +the Cache class affects only the cache and do not impact the backend database. + +Note that mixing cache sets with DB sets can create inconsistencies between +the database and the cache. You shouldn't do that unless you know what you're +doing. + +The classes use pickle to allow you to store and retrieve python objects in a +transparent way. To disable it, set .autopickle to False. + +Here is an example using the DB class: + +>>> import nmdb +>>> db = nmdb.DB() +>>> import nmdb +>>> db = nmdb.DB() +>>> db[1] = { 'english': 'one', 'castellano': 'uno', 'quechua': 'huk' } +>>> print db[1] +{'english': 'one', 'castellano': 'uno', 'quechua': 'huk'} +>>> db[(1, 2)] = (True, False) +>>> print db[(1, 2)] +(True, False) +>>> del db[(1, 2)] +>>> print db[(1, 2)] +Traceback (most recent call last): + File "", line 1, in + File "/usr/lib64/python2.5/site-packages/nmdb.py", line 74, in __getitem__ + raise KeyError +KeyError +""" + +import cPickle import nmdb_ll + class NetworkError (Exception): pass + class _nmdbDict (object): def __init__(self, db, op_get, op_set, op_delete): self.db = db - self.op_get = op_get - self.op_set = op_set - self.op_delete = op_delete + self.get = op_get + self.set = op_set + self.delete = op_delete + self.autopickle = True + + def add_server(self, port): + "Adds a server to the server pool." + rv = self.db.add_server(port) + if not rv: + raise NetworkError + return rv def __getitem__(self, key): + "d[k] Returns the value associated with the key k." + if self.autopickle: + key = str(hash(key)) try: - r = self.op_get(key) + r = self.get(key) except: raise NetworkError if not r: raise KeyError + if self.autopickle: + r = cPickle.loads(r) return r def __setitem__(self, key, val): - r = self.op_set(key, val) + "d[k] = v Associates the value v to the key k." + if self.autopickle: + key = str(hash(key)) + val = cPickle.dumps(val, protocol = -1) + r = self.set(key, val) if r <= 0: raise NetworkError return 1 def __delitem__(self, key): - r = self.op_delete(key) + "del d[k] Deletes the key k." + if self.autopickle: + key = str(hash(key)) + r = self.delete(key) if r < 0: raise NetworkError elif r == 0: @@ -40,15 +100,19 @@ return 1 def __contains__(self, key): + "Returns True if the key is in the database, False otherwise." + if self.autopickle: + key = str(hash(key)) try: - r = self.op_get(key) - except: - raise NetworkError + r = self.get(key) + except KeyError: + return False if not r: return False return True def has_key(self, key): + "Returns True if the key is in the database, False otherwise." return self.__contains__(key) @@ -63,7 +127,6 @@ db = nmdb_ll.connect(port) _nmdbDict.__init__(self, db, db.get, db.set, db.delete) - class SyncDB (_nmdbDict): def __init__(self, port = -1): db = nmdb_ll.connect(port) diff -rN -u old-nmdb/README new-nmdb/README --- old-nmdb/README 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/README 2007-01-08 23:58:05.000000000 -0300 @@ -17,6 +17,14 @@ directory, and is licensed individually. See the LICENSE file for more information. + +Documentation +------------- + +Most documentation is in the nmdb and libnmdb manpages, and inside the doc/ +directory. It is recommended that you read the User Guide (doc/guide.rst, or +the online version) to learn how to setup and use nmdb. + For additional documentation and resources, go to the project's website at http://auriga.wearlab.de/~alb/nmdb. diff -rN -u old-nmdb/TODO new-nmdb/TODO --- old-nmdb/TODO 2007-01-08 23:58:05.000000000 -0300 +++ new-nmdb/TODO 2007-01-08 23:58:05.000000000 -0300 @@ -6,9 +6,7 @@ Library: * Support multithreading and/with non-blocking API -* Automatic server selection Python: * Write test-cases to test the binding, the library and the server -* Document the module