10th Anniversary

10 years ago, Monty Widenius released MySQL 3.22.10.

That version was the first release of MySQL which included my contributions. I would submit a few more patches and write a couple of storage engines before I finally accepted Monty and David's 3rd or 4th job offer, over 4 years later; I cannot remember if the first offer, which occurred with great humor in December 1998 was in jest or serious. I have seen lots of stuff happen to MySQL over the last 10 years. Its user-base has grown far larger than I would have imagined 10 years ago - MySQL was not the first free database, nor was it the first opensource database. But it is friendly and has great people involved who are helpful and care about their users.

Time flies...

Re: For Internal Use Only - FAIL

In response to Monty Taylor's blog post ...

I am the culprit for the content of plugin.h which has the comment "the following declarations are for internal use only" which was introduced into the MySQL server codebase a couple of years ago.

Unfortunately, I had to work with the limitations of the C programming language and its preprocessor so to achieve a solution which behaved as if it had the strong typing like generics but working with simple C; I had to create a bunch of preprocessor defines of which are not intended for the plugin implementor to use but are the foundation for the macros which form part of the public API.

And at least I put in a comment telling people that they are for internal use. If you had access to the internal MySQL WorkLogs for this feature, it is perhaps one of the best well documented APIs for the server from the time of its introduction to the server codebase ... Historically before and since, new APIs are frequently added without any documentation nor comments.

Definitely not a "FAIL". It's working as designed.

Since your comment is in respect for the Drizzle project, feel free to scrap all that C defines and re-implement all of it nice and cleanly using C++ templates.

Isn't that the benefit of open-source? If you have an itch, scratch it.

The drizzled/plugin.h has mIxEd-CaSe defines. That ugliness appears to be a result from a global string search and replace.

Amira has landed

What or who is Amira?

Well... Many would know Amira is to David as My is to Monty.

Amira also has all the same letters as Maria, except that it is in PDP "middle-endian" byte order.

It is also the codename of a little known project while I was at MySQL: Soon after Oracle acquired Innobase OY, the powers that be at MySQL did not want to solely rely on InnoDB (or BerkeleyDB) as a transactional data store, mostly because they are both now owned by Oracle. So they shopped around, courted Jim Starkey, Monty wanted to do his own, or we can dig something out of the attic. Amira was the codename for the revival of the Gemini storage engine.

I admit, I have procrastinated for ages over this topic. Brain even put on his website the original sources and gave me a nudge... So I have now published the source to Launchpad.

Here it is at https://launchpad.net/amira

The plugin for MySQL 5.1 is far from complete and is the result of a couple of hours hacking I did yesterday. It builds but I can guarantee it will crash =)
It will take me a few weeks to fix it up to make it useful.

Alternatively, it should be possible to use the 5.0 version with only a few small patches into the server code. Unfortunately, I do not have a functional BitKeeper license to extract all the release versions of Amira which were internally built and tested - which is a shame. some of the earlier builds were quite stable and fast but at the time the project was canceled, there was a number of destabilizing changes done for performance improvements.
I'll have to try to remember what is was that I was doing in order to either complete the change or to back it out.

Closed-source revision control is not very useful.

MySQL plugins - the missing plug.in manual page.

A whole bunch of people have successfully written plugins for MySQL using the plug-in framework which first appears in MySQL 5.1. On the whole, the simple examples work great, as long as all your code is in one directory. Here is a helpful tip...

Inside the plug.in file, you can use MYSQL_PLUGIN_DIRECTORY_<FOO> as an alias to the directory in which your plug.in is located within, where FOO is the upper-case name of your plug-in.

Here is an example plug.in:

MYSQL_STORAGE_ENGINE(pdpse,,[PDP-SE Storage Engine], [Any old storage engine], [max,max-no-ndb]) MYSQL_PLUGIN_STATIC(pdpse, [libpdpse.la]) MYSQL_PLUGIN_DYNAMIC(pdpse, [ha_pdpse.la]) MYSQL_PLUGIN_ACTIONS(pdpse,[dnl AC_CONFIG_FILES(dnl MYSQL_PLUGIN_DIRECTORY_PDPSE/api/Makefile dnl MYSQL_PLUGIN_DIRECTORY_PDPSE/db/Makefile dnl MYSQL_PLUGIN_DIRECTORY_PDPSE/dbut/Makefile dnl MYSQL_PLUGIN_DIRECTORY_PDPSE/msgs/Makefile dnl MYSQL_PLUGIN_DIRECTORY_PDPSE/vst/Makefile dnl ) ])

Updated External Language Stored Procedures

I have merged the code from the mysql-5.1 launchpad repository which should put this repository up to the recent 5.1.28-rc version. You can examine the source repository here.
On a related note, had a brief email exchange with Eric Herman and so I hope that there will be time for him to do some work to complete the support for Java stored routines.

High performance stored procedures...

Wondering how interested people would be in this topic for MySQL Users Conference next year?

End of an era, (part 2)

I must admit, I was very surprised to read rumors of Monty quitting Sun. I haven't heard from him personally but if true I think it is sad but a decision that he has thoroughly thought about since Sun's acquisition of MySQL earlier this year.
I expect him to stay longer though: Perhaps mostly because when I chatted with him a couple of months ago, I had the impression that he intended to stay at least a year or until the Maria storage engine hit the v1.5 milestone.
My sense of foreboding I had in Orlando was not completely unfounded. I did try to be an optimist though - just see my blog posts from that time.

Late Spring Cleaning

Its facinating to see what a couple of years development and bugfixing work upon MySQL has unwittingly done to the free space on my main G5 PowerMac at home. I have captured a screenshot from GrandPerspective and as you can see, the largest file on my hard disk is the MirrorAgent.crash.log file, and all those pretty blocks on the left hand side of the window: They are all coredumps. A few swift commands at the console and I now have 154 GB of free disk space - two thirds free! Whoot!

So for all the MySQL Mac heads out there, check out your MirrorAgent crash log file and also the /cores subdirectory...

First week at Google.

Phew....

People ask what the first week at Google is like... I cannot find one word which can describe it but I'll try throwing together a few adjectives:

Exciting. Exhausting. Fun. Delicious.

The reason for the first adjective is obvious. The second adjective - simply because its like going back to school with stuff to learn. The third - I met a lot of interesting people and they are all fun to chat with. The last adjective - the on site cafés and cafeterias at the Mountain View campus serve tasty food.

I chatted with a few people there about databases and about MySQL. Looks like there shouldn't be any problems with using some of my 20% time for MySQL projects. I may even be able to find a few other likeminded individuals to tackle a few larger projects. Exciting stuff!

Now I have the weekend to unwind a little and prepare to do battle with Los Angeles traffic ... for when I drive to Santa Monica on Monday morning.

I am now at Google!

Yesterday, I became a Google employee.

I know that there has been quite a hiatus in my MySQL related activities but it wasn't all without good reason. The first proper holiday I have had in a few years.

Tomorrow afternoon, I hope to spend some time with Mark Callaghan ... It would be a nice break from the new-employee orientation and tutorials and hopefully a chance to meet more of my new colleagues.

Probably this coming weekend, I'll dust off the External Language SP work and perform some much needed tidy ups. It is a shame that Bazaar seems quite overtaxed by the MySQL codebase that doing a quick clone is no longer a possibility - specifying "hardlink" does speed it up but it still takes an age. For as much as we did complain at MySQL about BitKeeper, it at least did not often leave us waiting for unreasonable periods of time.

And before anyone asks.... No. I do not know what projects Google has in store for me. I expect I will find out soon enough.

Exciting times!

Video and Presentation now available

I have updated the Forge page for the External Language Stored Procedures, adding a PDF for the slides used in the presentation and a link to a video of the presentation for those people who missed it. It was quite unfortunate that at the Conference, there were two presentations occurring at the same time which discussed the topic of UDF/Stored Procedures so I am sure that there are people who would have liked to attend both events and had to choose.

Check out the updated page here:

http://forge.mysql.com/wiki/ProjectPage_External_Language_Stored_Procedures

After the MySQL Conference and Expo...

That week at the conference was a busy week for me - as busy as any developer meeting I have had with MySQL in recent years. I had a great time seeing old faces again and it was much like old times talking, chatting and coding with them. In particular, I spent much of my time with Patrick Galbraith, Eric Herman and Arjen Lentz.
Great progress was made: I worked with Patrick getting my ancient patch integrated into FederatedX so that it can support transactions. I also worked a bit with Arjen getting his OQGraph engine starting to answer a few simple queries. Of course, most importantly, I worked with Eric to finalize the plan for our presentation on "External Language Stored Procedures for MySQL". We also worked on getting the source repository hosted on LaunchPad and created a small team group to manage it. You can check it out here: Team Page
Now, just relaxing and chatting with people... and hopefully I will get a moment to get an updated patch out and to publish the presentation and demonstration notes. What I'd like to get done soon is to sync the External Lang repository with the current mysql-5.1 launchpad repository and maybe a small patch to support two-phase commits with FederatedX.

Impedance Mismatch

Amazing that there is such disconnect in service communications in these modern days when communications are seemingly instantaneous.

TriNet, the HR company which formerly managed MySQL Inc's personnel payroll and benefits, agrees that they have received my COBRA enrollment and first month's payments last Thursday, all information has been forwarded and that my Blue Shield coverage should continue without interruption. However, Blue Shield of California disagrees and says that coverage has terminated and even if they have received the information, it won't appear on their system for 30 days anyways.

*sighs*

This kind of makes things tricky for refilling prescriptions.

So much for 'uninterrupted service'.

Leaving MySQL, 20030908 - 20080321

Ask me a couple of months ago and I would have said that I was going to join Sun Microsystems after their acquisition of MySQL. Ask me two weeks ago and I would have said that I hope that things would work out.

At the end of this month, MySQL as a corporate entity would cease to exist and the MySQL employees become Sun Microsystems employees. Except for me and a small handful of people: Today, Friday 21 March 2008, is my final day working for MySQL. For various reasons, I felt I could not sign the agreements with Sun. This gives me an opportunity to go in a slightly different direction with my career.

Why have I only announced it on my final day? It was all quite sudden: Only on Tuesday I was informed that I would not be permitted to work through my notice period - which originally would have had me working until the middle of June. I had plans to get a couple of projects completed before I left and it would give me time for a smoother handover of tasks and perhaps a nice announcement a few weeks in advance. Who is now going to maintain the plug-ins code? I suppose it will now fall into someone's lap in a similar way how it fell into my lap: Somewhat haphazardly and unplanned. Of course, that happened soon after InnoDB was purchased by Oracle and Martin Mickos, made an announcement about plug-in storage engines being the way of the future - someone had to work on it to make it happen.... **plonk**
Still, I hate leaving things partially completed. Just makes things untidy for someone else to take over. Not that I have much choice under the current circumstance. I will resume being a member of the MySQL community and will stay in touch with MySQL, contribute a few patches. I told Jeffrey that there needs to be a better process for community to interact with the MySQL code - he said that something is being worked out.

So many things going to happen soon too - next month, April, has the MySQL User Conference and Expo in Santa Clara - I hope that our presentation on External Language Stored Procedures for MySQL is informative and well received. In May, I have a much deserved vacation which was booked last year - I had plenty of accrued vacation. Those plans are not going to change now - so I suppose I will start my new job in June.

Oh well, end of a 4 years 6 month chapter... also end of the MySQL-as-a-small-upstart-company era too. I wish all my former colleagues, including those which have now joined Sun, all the best for the future. I hope that Sun continues the great MySQL traditions. keep growing their business, keep MySQL as a fun product to use.

Aggregate Functions with Perl stored procedures.

I found myself thinking this evening about how someone could set about writing aggregate functions using Perl and on an idle Google search, came across this webpage entitled User-defined Aggregate Functions in DB2 Universal Database.

I wondered if a similar technique could be applied for our implementation of External Language Stored Procedures for MySQL. It turns out that the answer is: Yes.

Suppose you had the following Perl declarations in a Perl module:

our %summary=(); sub aggregate_add($$) { my ($value,$group)= @_; if (defined $value) { $summary{$group}= {value=>0.0, value2=>0.0, count=>0} if !defined $summary{$group}; my $scalar= scalar $value; $summary{$group}{value}+= $scalar; $summary{$group}{value2}+= $scalar * $scalar; $summary{$group}{count}++; } return $value; } sub aggregate_result($$) { my ($value,$group)= @_; return undef if !defined $summary{$group}; my $count= $summary{$group}{count}; my $average= $summary{$group}{value} / $count; my $sqavg= $summary{$group}{value2} / $count; my $variance= $sqavg - ($average * $average); my $stddev= sqrt $variance; return sprintf("count=%d avg=%0.3f var=%0.3f std=%0.3f", $count, $average, $variance, $stddev); }

Now we need to test this little piece of Perl magic by starting up a client session...

mysql> CREATE TABLE t1 (grp INT, a DOUBLE);
Query OK, 0 rows affected (0.00 sec)

mysql> INSERT INTO t1 VALUES (1,1), (2,2), (2,3), (3,4), (3,5), (3,6);
Query OK, 6 rows affected (0.00 sec)
Records: 6  Duplicates: 0  Warnings: 0

mysql> CREATE FUNCTION test.agg_result(value DOUBLE, grp INT) RETURNS CHAR(128)
    -> LANGUAGE Perl NO SQL EXTERNAL NAME 'Foo::aggregate_result';
Query OK, 0 rows affected (0.01 sec)

mysql> CREATE FUNCTION test.agg_add(value DOUBLE, grp INT) RETURNS DOUBLE
    -> LANGUAGE Perl NO SQL EXTERNAL NAME 'Foo::aggregate_add';
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT COUNT(a),
    ->        CAST(AVG(a) AS DECIMAL(7,3)) 'AVG',
    ->        CAST(VARIANCE(a) AS DECIMAL(7,3)) 'VAR',
    ->        CAST(STD(a) AS DECIMAL(7,3)) 'STD',
    ->        test.agg_result(MAX(test.agg_add(a,grp)),grp) 'TEST'
    ->        FROM t1 GROUP BY grp;
+----------+-------+-------+-------+---------------------------------------+
| COUNT(a) | AVG   | VAR   | STD   | TEST                                  |
+----------+-------+-------+-------+---------------------------------------+
|        1 | 1.000 | 0.000 | 0.000 | count=1 avg=1.000 var=0.000 std=0.000 |
|        2 | 2.500 | 0.250 | 0.500 | count=2 avg=2.500 var=0.250 std=0.500 |
|        3 | 5.000 | 0.667 | 0.816 | count=3 avg=5.000 var=0.667 std=0.816 |
+----------+-------+-------+-------+---------------------------------------+
3 rows in set (0.04 sec)

Whoot! It actually works! Note that this only works because the Perl instances are not shared between multiple threads and so all the calls to those subroutines and their variables exist within the calling thread only. This avoids any nasty synchronization headaches. I would imagine that it would not be much more difficult to do something similar with the Java plugin except that the global statistics variable would have to be a ThreadLocal instance to avoid multiple threads conflicting with each other.

I guess I shall just have to include it as a test case; at least until a concrete method to declare aggregate stored functions arrives. I would imagine that such a declaration would name existing stored procedures in order to construct the aggregate function.

Something to think about for the future.

(An unrelated side note, March is Endometriosis Awareness Month. Please spend a thought for women who suffer from this disabilitating disease... Thanks)

Here, there be dragons...

Still shell-shocked from the announcement of Sun's intent to acquire MySQL, I was among the many at Martin's presentation for whom his words seemed to miss our ears and whose eyes failed to register what was on the slides before us. A sea of silence. Martin quite nerviously continued on while, for me, I was stunned, expecting an IPO announcement and thinking that what was before me was a hypothetical situation being shared.

It was very real.

Now that I have had a couple of days to allow the news to embed, I can reflect upon the situation more rationally: Out of all the big names in the IT industry, Sun is perhaps the best and only choice for which the purchase would not be for the brand. MySQL has a very real chance of sustained survival in Sun's ecosystem. Talking to many collegues, I think the outlook is generally very positive except with a small number of reservations revolving around IP issues. As to how I personally feel: Still a little stunned but I think that for the ongoing future survival for MySQL as a product and as a community, being acquired by Sun is the best possible scenerio.

As MySQL is now set on a course into terra incognita, being swallowed whole by Sun, MySQL's staff will have to navigate with trepidation in our new world. Where MySQL is a small company full of smart individuals, there is a fear among some that their voices will be lost among the ocean of Sun's talented.

Time will tell.