Repairing a low-quality picture with Photoshop
Abstract
Making photos to share with friends is a popular thing to do, but often the quality of the picture is poor. The colorbalance is off, the picture lacks contrast or the picture is distorted. In some cases it is worth retouching a photo. This post qualitatively shows how.
Taking photos from touristic hotspots. A very popular activity for the self-obsessed traveler. Most people own a photo camera, as part of their phone or standalone. Most people also find themselves interesting, and like to share their whereabouts with others. Through Facebook, Twitter or other beloved social media people express themselves by uploading photos. A picture is worth a thousand words - The Dictionary of Clichés. But as much as vanity makes believe, from an esthetic point of view these pictures are usually of poor quality (which also applies for my own pics). Now I don't have to be a professional photographer to draw this conclusion. Take for example this picture.
It is a great photo, through its content. A very nice view of the lower part of Manhatten, New York City. The photo may even have some sentimental value to some. No problem. But esthetically, it is terrible. The horizon is slightly twisted. The air is very grainy. There is a distracting sun reflection to be seen on one of the buildings. And worse of all: the colors are very dull and mat. Most of these flaws are due to poor camera quality. Can these problems be repaired? Yes. Photoshop is our friend.
Javascript minifier
Abstract
For dynamic websites using many AJAX features, Javascript files often become large. We can reduce the file size - without changing functionality - with Javascript Minifier.
It is common pratice to write Javascript (or in fact any sort of scripting language) in such a way that it is readable. Readability is improved by comments, indentation, proper variable name declaration, etc. But this makes the source become larger than necesary. For most languages this is not problem, as a compiler translates the code instructions into a machine-readable language. For Javascript however, having large source files means more traffic. Let's reduce this.
There are many Javascript compressors available, such as Dojo Shrinksafe, Packer and YUI Compressor. But I prefer to use good old Javascript Minifier. Two reasons: 1) it is safe (it does not change any functionality, because it uses very simple shrink techniques) and 2) it is conveniently compilable from the C (so no Java required).
The source is available from Github and compilable with gcc -o jsmin jsmin.c. In this case, I am using jsmin for Wiziwig. On the development server, the non-minified versions of the Javascript files run. Upon pushing a new release to the production servers, jsmin is launched (on production server):
jsmin < site.js > site-min.js mv site-min.js site.js
The result looks like this.
Smarty Memcached: high performance template caching
Abstract
Smarty is a powerfull template system. It provides a large set of dynamic template capabilities. It can also provide caching - file based caching. That of course is not fast enough, as it can cause IO overhead. Mecached once again comes to the rescue.
I have only started using Smarty recently. I quickly discovered its benifits. You can create very compact templates, with dynamic behavior. Smarty does not just stop with having the capability to insert PHP variables into the template. It allows for integrating complex structures into your template, such as for, foreach and if/else statements.
Smarty also comes with caching capabilities. If cache is enabled, Smarty will store compiled templates on disk, and serve future requests from that cache, rather than having to compile the template again. On php side, this also allows you to prevent having to fetch the same data from the database again, and having to perform the same redundant calculations all over again - all for the same result.
Matlab Fast SOAP
Abstract
Matlab comes with intrinsic SOAP message handling functions. These rely on a DOM approach. While flexible and robust, the DOM approach has undesirable scaling for large XML documents. To avoid this, I have written a set of replacement functions.
For Simple Object Access Protocol (SOAP) document handling, Matlab has three embedded functions that all rely on a Java DOM object:
- createSoapMessage(): This function takes data structured in a cell object, and places this data in the SOAP object. For every insertion, it has to iterate through all previously inserted elements, to find the right position. It is not capable of building a set of elements and inserting all these elements at once into the SOAP object.
- callSoapService(): This function takes the generated SOAP object, converts the object into a Java class string and sends out the bytes to the web service. Upon a successful upload, the web service will give its response - whatever that may be. After a finished transaction, the callSoapService function will store the resulting document.
- parseSoapResponse(): Take the response and parse all body elements into a new DOM object. This function, as well as createSoapMessage(), will individually select an element and insert it into the object. For every next element, it has to iterate both through document and object.
GSD: the 'Getting Stuff Done' to-do manager
Abstract
I'm often working on a lot of projects at the same time. I tend to lose track of all the things that I have to do within a single project. So I make plain text to-do lists. This is great, but also a little cumbersome, especially with large to-do lists. Now there's GSD.
GSD, written by fellow student Jason Graham, is an abbreviation for Getting Stuff Done. GSD is a command line shell script, saving its data in ~/.gsd. It creates and manages to-do lists with very intuitive commands. For example gsd add "Give Patrick a call to tell him he's a douche" will make sure you will never forget to offend Patrick - whoever that may be. And once you've made that unpleasent phone call, you can mark this to-do item as completed with gsd edit -s C id.
Through simplicity, GSD becomes genius. And because of that, I tried to help Jason out a little bit with a few bugs. Mac OSX is now tested and supported, GSD throws a nice info message if you fail to provide proper arguments and output is nicely aligned.
Try out GSD for yourself. You may download the latest version at the Github repository.
Creating and restoring MySQL hotcopy backups
Abstract
With MySQL hotcopy we can create backups by copying the binary database files. This is fast and clean. We still need to perform table locks, but this goes much faster. How to do create backups, and how to restore them?
mysqlhotcopy is a Perl script to create MySQL backup. In contrast with the mysqldump program, mysqlhotcopy does not generate any SQL formatted output. All it does is force a lock on a table, then copy the raw table files, and release the lock. Mysqlhotcopy only works on MyIsam's, and only on a local machine.
To create a mysqlhotcopy backup, I use the command
/usr/local/mysql/bin/mysqlhotcopy --allowold -user=$DBUSER -password=$DBPASS $DBNAME $backupdir
With of course the right shell variables set. This command will copy a bunch of .frm, .MYD and .MYI files to the specified directory. You can now do with this whatever you want. I'm packing all the files in a tar.bz2 with:
tar -C $backupdir/ -cjf $backupdir/$DBNAME-1.tar.bz2 $DBNAME
After packing it, I deploy the backup to our backup servers, and bring it into the backup cycle.
Restoring
This is all fairly straight forward. The problem starts when you have to restore these files. You cannot run an SQL insert on a running database as you would do with a mysqldump .sql backup. You will have to switch off the database. Then extract your backup to the MySQL data directory, usually something like /var/lib/mysql. Once this is done, you will have to rebuild the database indices. Do this with:
myisamchk -rq *.MYI
You should execute this command from within your database file directory, so in our case: /var/lib/mysql/$DBNAME. That's all there is to it.
Matlab project on Github
Abstract
Finally, I created my first public repository on Github. Read where to find it, and how to keep track of my stuff.
My internship here at the Johns Hopkins University has me working on a Matlab software package. This package, that will be available freely to the public, is still in development. So far, I have been developing it without keeping track of any changes. I have not been using any VCS.
So now, I've created a public repository on Turbmat. Check it out at Github. You can follow me, or my Turbmat code. You can create a fork and if you want, contact me to send in any contributions.
Stage JHU in Baltimore #11
Abstract
Afgelopen dinsdag heb ik opnieuw een presentatie gegeven over mijn werk. In dit bericht weer een slideshare van mijn presentatie, met wat uitleg.
De presentatie kun je bekijken als je even op "Read on.." klikt, of op de titel van dit bericht.
Nieuw in mijn project zijn drie onderdelen: energie spectra van de turbulente stroming, PDF's van snelheidscomponenten en druk, en PDF's van snelheidsincrementen.
Met name die laatste kostte tamelijk veel werk, omdat er een slimme manier gevonden moet worden om zo veel mogelijk data binnen te krijgen. Dit is nodig om een goede nauwkeurigheid te krijgen in de gebieden die ver van het gemiddelde liggen in de PDF.
Ik haal willekeurig gepositioneerde en georiënteerde blokken op uit de database (zie slide 12). Deze blokken zijn 32x128x512 fysieke databasepunten groot, maar worden opgevraagd door 32x32x32 regelmatige verdeelde punten. Om een goed geconvergeerd resultaat te krijgen is het nodig om zo'n 20 blokken op te halen. In totaal ruim 650 duizend punten.
De resultaten, slide 13 en 14, komen goed overeen met de theorie.