Published article
Abstract
Link to article on which I am co-auther
The article on which I am a co-auther, and which covers a part of the research conducted at JHU in Baltimore, has been pre-published. The article will soon appear in Journal of Turbulence. Find the article on the JHU Turbulence website, under the title Studying Lagrangian dynamics of turbulence using on-demand fluid particle tracking in a public turbulence database.
Tunneling past VPN
Abstract
VPN servers often are slow, as they form a converging funnel for data traffic. Tunneling your traffic around the VPN server is a great call here.
Most companies or organizations have their computational services in a private network. The only way to access these services - such as data storage or remote desktop - is to connect through a VPN server. The VPN connection emulates the existence of a private network attached to your computer, as if you were at work. Very nice if you need to use all available resource within the private network.
Major drawback is speed. Often, the VPN server is a single physical point. Indeed, a single point of failure, as well as a funnel for your and other people's data. The limitations of the VPN connection are directly related to the performance of the VPN server - its NIC, CPU and memory.
There is a way around this. The idea is simple. If you cannot directly connect to your server, then let your server connect directly to you. This tutorial is based on the power of SSH, and through that it is limited to *nix systems. Yet, I am sure there are options for Windows too.
Repairing a low-quality picture with Photoshop
Abstract
Making photos to share with friends is a popular thing to do, but often the quality of the picture is poor. The colorbalance is off, the picture lacks contrast or the picture is distorted. In some cases it is worth retouching a photo. This post qualitatively shows how.
Taking photos from touristic hotspots. A very popular activity for the self-obsessed traveler. Most people own a photo camera, as part of their phone or standalone. Most people also find themselves interesting, and like to share their whereabouts with others. Through Facebook, Twitter or other beloved social media people express themselves by uploading photos. A picture is worth a thousand words - The Dictionary of Clichés. But as much as vanity makes believe, from an esthetic point of view these pictures are usually of poor quality (which also applies for my own pics). Now I don't have to be a professional photographer to draw this conclusion. Take for example this picture.
It is a great photo, through its content. A very nice view of the lower part of Manhatten, New York City. The photo may even have some sentimental value to some. No problem. But esthetically, it is terrible. The horizon is slightly twisted. The air is very grainy. There is a distracting sun reflection to be seen on one of the buildings. And worse of all: the colors are very dull and mat. Most of these flaws are due to poor camera quality. Can these problems be repaired? Yes. Photoshop is our friend.
Javascript minifier
Abstract
For dynamic websites using many AJAX features, Javascript files often become large. We can reduce the file size - without changing functionality - with Javascript Minifier.
It is common pratice to write Javascript (or in fact any sort of scripting language) in such a way that it is readable. Readability is improved by comments, indentation, proper variable name declaration, etc. But this makes the source become larger than necesary. For most languages this is not problem, as a compiler translates the code instructions into a machine-readable language. For Javascript however, having large source files means more traffic. Let's reduce this.
There are many Javascript compressors available, such as Dojo Shrinksafe, Packer and YUI Compressor. But I prefer to use good old Javascript Minifier. Two reasons: 1) it is safe (it does not change any functionality, because it uses very simple shrink techniques) and 2) it is conveniently compilable from the C (so no Java required).
The source is available from Github and compilable with gcc -o jsmin jsmin.c. In this case, I am using jsmin for Wiziwig. On the development server, the non-minified versions of the Javascript files run. Upon pushing a new release to the production servers, jsmin is launched (on production server):
jsmin < site.js > site-min.js mv site-min.js site.js
The result looks like this.
Smarty Memcached: high performance template caching
Abstract
Smarty is a powerfull template system. It provides a large set of dynamic template capabilities. It can also provide caching - file based caching. That of course is not fast enough, as it can cause IO overhead. Mecached once again comes to the rescue.
I have only started using Smarty recently. I quickly discovered its benifits. You can create very compact templates, with dynamic behavior. Smarty does not just stop with having the capability to insert PHP variables into the template. It allows for integrating complex structures into your template, such as for, foreach and if/else statements.
Smarty also comes with caching capabilities. If cache is enabled, Smarty will store compiled templates on disk, and serve future requests from that cache, rather than having to compile the template again. On php side, this also allows you to prevent having to fetch the same data from the database again, and having to perform the same redundant calculations all over again - all for the same result.
Matlab Fast SOAP
Abstract
Matlab comes with intrinsic SOAP message handling functions. These rely on a DOM approach. While flexible and robust, the DOM approach has undesirable scaling for large XML documents. To avoid this, I have written a set of replacement functions.
For Simple Object Access Protocol (SOAP) document handling, Matlab has three embedded functions that all rely on a Java DOM object:
- createSoapMessage(): This function takes data structured in a cell object, and places this data in the SOAP object. For every insertion, it has to iterate through all previously inserted elements, to find the right position. It is not capable of building a set of elements and inserting all these elements at once into the SOAP object.
- callSoapService(): This function takes the generated SOAP object, converts the object into a Java class string and sends out the bytes to the web service. Upon a successful upload, the web service will give its response - whatever that may be. After a finished transaction, the callSoapService function will store the resulting document.
- parseSoapResponse(): Take the response and parse all body elements into a new DOM object. This function, as well as createSoapMessage(), will individually select an element and insert it into the object. For every next element, it has to iterate both through document and object.
GSD: the 'Getting Stuff Done' to-do manager
Abstract
I'm often working on a lot of projects at the same time. I tend to lose track of all the things that I have to do within a single project. So I make plain text to-do lists. This is great, but also a little cumbersome, especially with large to-do lists. Now there's GSD.
GSD, written by fellow student Jason Graham, is an abbreviation for Getting Stuff Done. GSD is a command line shell script, saving its data in ~/.gsd. It creates and manages to-do lists with very intuitive commands. For example gsd add "Give Patrick a call to tell him he's a douche" will make sure you will never forget to offend Patrick - whoever that may be. And once you've made that unpleasent phone call, you can mark this to-do item as completed with gsd edit -s C id.
Through simplicity, GSD becomes genius. And because of that, I tried to help Jason out a little bit with a few bugs. Mac OSX is now tested and supported, GSD throws a nice info message if you fail to provide proper arguments and output is nicely aligned.
Try out GSD for yourself. You may download the latest version at the Github repository.
Creating and restoring MySQL hotcopy backups
Abstract
With MySQL hotcopy we can create backups by copying the binary database files. This is fast and clean. We still need to perform table locks, but this goes much faster. How to do create backups, and how to restore them?
mysqlhotcopy is a Perl script to create MySQL backup. In contrast with the mysqldump program, mysqlhotcopy does not generate any SQL formatted output. All it does is force a lock on a table, then copy the raw table files, and release the lock. Mysqlhotcopy only works on MyIsam's, and only on a local machine.
To create a mysqlhotcopy backup, I use the command
/usr/local/mysql/bin/mysqlhotcopy --allowold -user=$DBUSER -password=$DBPASS $DBNAME $backupdir
With of course the right shell variables set. This command will copy a bunch of .frm, .MYD and .MYI files to the specified directory. You can now do with this whatever you want. I'm packing all the files in a tar.bz2 with:
tar -C $backupdir/ -cjf $backupdir/$DBNAME-1.tar.bz2 $DBNAME
After packing it, I deploy the backup to our backup servers, and bring it into the backup cycle.
Restoring
This is all fairly straight forward. The problem starts when you have to restore these files. You cannot run an SQL insert on a running database as you would do with a mysqldump .sql backup. You will have to switch off the database. Then extract your backup to the MySQL data directory, usually something like /var/lib/mysql. Once this is done, you will have to rebuild the database indices. Do this with:
myisamchk -rq *.MYI
You should execute this command from within your database file directory, so in our case: /var/lib/mysql/$DBNAME. That's all there is to it.