memcached MATLAB

UPDATE: 2015-03-10 - I was able to get spymemcached to work, which is very nice because it is

• very fast
• the first java client listed on the memcached wiki
• it is by the same folks who ported memcached to Windows
• has many, many options
• the most current, updated to 2.11.6 this January-2015

.

Introduction

memcached is great for sending objects from process to process. I was able to use memcached to set and retrieve objects from two different instances of MATLAB. Here I used both Java and .NET libraries.

Downloads

I compiled Windows binaries of memcached-client libraries for both the Java and .NET using Java-1.7 and Visual Studio 2010 and zipped them up here.

The memcached server was compiled by Trond Norbye from Couchbase (previously Membase previously Northscale) for  32-bit and 64-bit Windows OS. See below how to run it as a service using the Python PyWin32 package.

.NET

Details

Memcached server version 1.4.5 patched for 64-bit Windows by Trond Norbye from Couchbase (previously Membase previously Northscale).
Memcached .NET client: Memcached.ClientLibrary port of com.meetup.memcached from sourceforge.

Build

Microsoft Visual Studio 2010 C# - Release/AnyCPU configuration. Since the original project files were for MSVC-8, MSVC-10 started an conversion wizard that first backed up the previous project files and then created a new MSVC-10 project files including a new solution file. Then I only build the clientlib_2.0 project from the MSVC-10 IDE GUI using build from the menu, and voila a new dll.

Usage

1. Start memcached server

C:\> C:\path\to\memcached.exe -vv

2. Do this on both MATLAB instances

>> asm = NET.addAssembly('C:\full\path\to\Memcached.ClientLibrary.dll')
>> pool = Memcached.ClientLibrary.SockIOPool.GetInstance()
>> pool.SetServers({'127.0.0.1:11211'})
>> pool.Initialize
>> mc = Memcached.ClientLibrary.MemcachedClient()

3. MATLAB instance #1

>> mc.Set('name','mark')
    1

4. MATLAB instance #2

>> mc.Get('name')
mark

Java

MATLAB and memcached also works perfectly with the original meetup.com Java version that the .NET version was ported from. The commands are exactly the same as the .NET version.

Source:

com.meetup.memcached

Build:

There are two branches in Greg Whalin's GitHub repository: master and performance. Performance is more recent and uses a Schooner client which they benchmark against the older meetup.com (Whalin) client as well as spymemcached (Couchbase/Membase) and xmemcache (see below), and on the wiki pages and in a powerpoint presentation in the doc folder they report that the performance branch is allegedly faster.

You can build the performance branch either from the repo, or download a zip file of the Memcached-Java-Client-3.0.0 release from the downloads page. There are several required libraries to make this run in the lib folder of the performance branch also included in the zip-file or tar-ball.

I was able to build this using jdk-7 by ...

1. extracting the file,
2. making a new bin folder at the same level as lib and src,
3. then navigating to src\main\java
4. and executing the following:

C:\> "C:\Program Files\Java\jdk1.7.0_55\bin\javac.exe" -verbose -cp ..\..\..\lib\commons-pool-1.5.6.jar;..\..\..\lib\slf4j-api-1.6.1.jar;..\..\..\lib\slf4j-simple-1.6.1.jar -sourcepath com\schooner\MemCached\ -source 1.6 -target 1.6 -d ..\..\..\bin com\whalin\MemCached\*.java com\schooner\MemCached\*.java com\schooner\MemCached\command\*.java

5. Then back up to the new bin folder an executing this:

c:\> "C:\Program Files\java\jdk1.7.0_55\bin\jar.exe" -cvf Memcached-Java-Client.jar com\

There are also newer jar files (3.0.2) from the Maven repository. All of the newer performance branch versions (3.0.0 and 3.0.2) work almost exactly like the old meetup.com client, but the package is named com.whalin.MemCached instead. Also now you will also have to add commons-pool-1.5.6.jar, slf4j-api-1.6.1.jar and slf4j-simple-1.6.1.jar to your MATLAB Java classpath using javaaddpath(). Finally the logger output is cleaned up so that you don't have to set up any logger properties as in the master branch. In fact if you do want to set up logging properties, the new performance branch doesn't use apache.org.log4j it uses slf4j simple logger.

>> javaaddpath('C:\full\path\to\commons-pool-1.5.6.jar')
>> javaaddpath('C:\full\path\to\slf4j-api-1.6.1.jar')
>> javaaddpath('C:\full\path\to\slf4j-simple-1.6.1.jar')
>> javaaddpath('C:\full\path\to\Memcached-Java-Client-3.0.2.jar')
>> import com.whalin.MemCached.MemCachedClient
>> import com.whalin.MemCached.SockIOPool
>> pool = SockIOPool.getInstance;
>> pool.setServers({'127.0.0.1:11211'})
>> pool.initialize
>> mc = MemCachedClient;
>> pool.getAliveCheck
    1
>> mc.set('name','mark')
    1
>> mc.get('name')
mark
>> pool.shutDown
>> pool.getAliveCheck
    0

You can also build the master branch from the Github source.

$ /c/Program\ Files/Java/jdk1.7.0_55/bin/javac.exe -verbose -classpath ../lib/log4j.jar -sourcepath com/meetup/memcached/ -source 1.6 -target 1.6 -d ../bin/ com/meetup/memcached/*.java
$ /c/Program\ Files/Java/jdk1.7.0_55/bin/jar.exe -cvf MeetupMemcached.jar com/

In the master branch, the Apache logger is not cleaned up for the non-bench/test sources (with isErrorEnabled(), etc. see jlog4 documentaion), so you always get this error message:

log4j:WARN No appenders could be found for logger (com.meetup.memcached.SockIOPool).
log4j:WARN Please initialize the log4j system properly.

Because there are log calls in the MemcachedClient and SockIOPool files that never have BasicConfiguration.configure() or set any jlog4.<properties> like the appender hence the error. Of course they never meant for log messages to be displayed when deployed, so wrapping with isErrorEnabled() like the .NET version does would be better. In order to get rid of that message and output logging messages to a file configure the Apache.org log4j logger yourself by creating a FileAppender with the default layout and then configuring the BasicConfigurator.

>> fileAppender_withDefaultLayout = org.apache.log4j.FileAppender(org.apache.log4j.PatternLayout,'memcached.log')
fileAppender_withDefaultLayout =
org.apache.log4j.FileAppender@60f585a2
>> org.apache.log4j.BasicConfigurator.configure(fileAppender_withDefaultLayout)

An optional boolean 3rd argument appends or overwrites the log file. Relative paths seem to be working so far, but full absolute path might be better.

Usage:

First start memcached server, then start sock IO pools and memcached clients on both matlab instances.

>> fileAppender_withDefaultLayout = org.apache.log4j.FileAppender(org.apache.log4j.PatternLayout,'memcached.log')
>> org.apache.log4j.BasicConfigurator.configure(fileAppender_withDefaultLayout)
>> javaaddpath('C:\full\path\to\MeetupMemcached.jar')
>> pool = com.meetup.memcached.SockIOPool.getInstance()
>> pool.setServers({'127.0.0.1:11211'})
>> pool.initialize
>> mc = com.meetup.memcached.MemcachedClient
>> mc.set('name','mark') % on 1st instance/process
    1
>> mc.get('name') % on 2nd instance/process
mark

Other clients

• xmemcached - Github:killme2008, Google Code project and Maven repo
The Xmemcached client works well. It is the most recently updated. There are jar files on the releases page of the GitHub repo. Here is an example from the 2.0.0 release from this April. The example is from the google.code wiki User Guide in English.

>> javaaddpath('C:\full\path\to\xmemcached-2.0.0.jar')
>> addr = net.rubyeye.xmemcached.utils.AddrUtil.getAddresses('localhost:11211')
addr =
[localhost/127.0.0.1:11211]
>> builder = net.rubyeye.xmemcached.XMemcachedClientBuilder(addr)
builder =
net.rubyeye.xmemcached.XMemcachedClientBuilder@7ef6a26
>> mc = builder.build()
log4j:WARN No appenders could be found for logger (net.rubyeye.xmemcached.XMemcachedClient).
log4j:WARN Please initialize the log4j system properly. 
mc = 
net.rubyeye.xmemcached.XMemcachedClient@275e6ce5
>> mc.set('name',0,'mark')
ans =
     1
>> mc.get('name')
ans =
mark

To get rid of the logger message you will have to download SLF4J and add slf4j-api-X.Y.Z.jar, slf4j-simple-X.Y.Z.jar and slf4j-log4j-X.Y.Z.jar to your path first, where X.Y.Z is the SLF4J version, 1.7.7 as of 2014-08-27. Then configure log4j as above for com.meetup.memcached.

>> fileAppender_withDefaultLayout = org.apache.log4j.FileAppender(org.apache.log4j.PatternLayout,'memcached.log')
fileAppender_withDefaultLayout =
org.apache.log4j.FileAppender@60f585a2
>> org.apache.log4j.BasicConfigurator.configure(fileAppender_withDefaultLayout)

• Enyim - NuGet and Github:enyim
I couldn't get this assembly to load as built, I always got an "Strong Name Validation Failed" error, perhaps because I am on a 64-bit machine, but using MSVC express.
• spymemcached - Github:couchbase (was dustin), Google Code project and Maven repo
UPDATED: 2015-03-11 I finally get this to work. I used AddrUtil to make building the client easy, similar to xmemcached. Everything just works. Download the latest jar from Maven Central, from GitHib releases page or a somewhat older jar from the google code download page.

>> javaaddpath('C:\Users\mmikofski\Downloads\spymemcached-2.10.3.jar')
>> import net.spy.memcached.AddrUtil
>> import net.spy.memcached.MemcachedClient
>> mc = net.spy.memcached.MemcachedClient(net.spy.memcached.AddrUtil.getAddresses('localhost:11211'))

2015-03-10 22:24:44.508 INFO net.spy.memcached.MemcachedConnection:  Added {QA sa=localhost/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
 
mc =
 
Failure Mode: Redistribute, Hash Algorithm: NATIVE_HASH Max Reconnect Delay: 30, Max Op Timeout: 2500, Op Queue Length: 16384, Op Max Queue Block Time10000, Max Timeout Exception Threshold: 998, Read Buffer Size: 16384, Transcoder: net.spy.memcached.transcoders.SerializingTranscoder@2a744425, Operation Factory: net.spy.memcached.protocol.ascii.AsciiOperationFactory@3725eee9 isDaemon: false, Optimized: false, Using Nagle: false, ConnectionFactory: DefaultConnectionFactory
 
2015-03-10 22:24:44.514 INFO net.spy.memcached.MemcachedConnection:  Connection state changed for sun.nio.ch.SelectionKeyImpl@7deebece

>> mc.set('name',0,'mark')

ans =
 
net.spy.memcached.internal.OperationFuture@6126121f

>> mc.get('name')

ans =

mark

Seriously, could this be any easier?

A lot of people have asked how to test if spymemcached connection is alive. This is because spymemcached will keep trying to connect or complete it's task even if it can't connect to the memcached server, which is nice if the server goes for a second then comes back up. There solution is to use a Future async task by calling their .asyngGet() method. Another way is to test the socket using java.net.Socket(inetAddress) which will raise an exception if it fails to connect.

InetSocketAddress addr = new java.net.InetSocketAddress("localhost", 11211);
try {
    Socket sock = new java.net.Socket(addr.getAddress, addr.getPort);
} except (java.net.ConnectException) {
...;
} else {
sock.close();
}

memcached server as Windows service

This is pretty easy using Python PyWin32 win32service.