How I Used Query Analysis to Speed up My Applications

I’ll sort out the scripts and bits and pieces I use once I get back to the office next week, for those people waiting for that material.

I obviously have my four presentations to get through, and there will be plenty of other stuff going on at the conference both in terms of other presentations, the show floor, the booths, and the Birds of a Feather sessions (which are terrific fun).

But mostly, I see the UC2009 as the best opportunity to meet up with both other MySQL/Sun people, our customers, and the community at large and talk about the thing we are all passionate about – MySQL and the technologies surrounding it.

And I don’t care if that sounds goofy. Nothing beats talking to face to face with like minded individuals, and one thing that MySQL – and Sun – seems to engender is an extreme passion about the products we deliver that I don’t always see in other products or companies. That means the talk is always interesting and intelligent, and, when it needs to be, good fun.

To anybody attending, welcome to the fun and enjoy the experience. To those not coming, you really are missing something.

First up for me, on Monday afternoon, is my tutorial: Scale Up, Scale Out, and High Availability: Solutions and Combinations.

What will be doing?

Very simply: Looking at every potential solution for maximum scalability and availability for your database environment.

If you are attending, be prepared to:

• Expand your mind as we think about scaling up.
• Expand your horizons as we think about scaling out.
• Divide and conquer, as we think about high-availability.

We’re not not hands on in this session – but I will expect you to be brains on!

I write my memories in my Moleskines, not on a day to day basis, but whenever anything significant is happening in my life, or when I’m traveling about the world for business or pleasure, and for me that’s a vital part of me recording my memories, despite my otherwise excellent recollection skills.

Hundreds of years ago Samuel Pepys and many others did the same thing – diarized what was going on in their lives and the world around them, and it became part of the content that records our world history. I know not every blog is that interesting (my own including), but as the title says, what goes around, comes around – we are just carrying on the tradition of those great diarists that came before us.

Despite having computers, organization tools, and email, there is something decidedly comforting about writing, by hand, into a physical notebook.

Of course, you write in pencil so that the contents don’t smudge, and somehow that just feels even better.

I cover both choosing the best release version, using tricks like mtmalloc (the threaded malloc library) before moving on to UFS and ZFS tricks, using DTrace and MySQL Cluster and Sun Cluster.

Despite this I only 7 out of 10, but I still think that’s a respectable score.

Myth:

MySQL Documentation is written by the developers.

Reality

MySQL Documentation is written by a dedicated team of writers with help and input from the developers. There are four main writers, Paul DuBois, Tony Bedford, Jon Stephens, and MC Brown (me!), plus our Team Lead, Stefan Hinz.

All the documentation staff are employed full time for the sole purpose of writing documentation. Sure, some of us get involved in other things too, but that’s basically the nature of the job. Some of us simply cannot help ourselves.

Myth

Docs team members are just writers and have no technical expertise.

Reality

It’s tempting to come back with a rude response to this one, but it is a comment I heard from someone at a conference. The reality is that all of us have some technical background, unsurprisingly often with MySQL. Some of us have expertise elsewhere too. Speaking only for myself, go look at MCslp.com for more info. If you want details, feel free to ask, but know that this myth is definitely busted.

Myth

The documentation is updated very rarely.

Reality

Our main tree, mysqldoc, is publicly available, and if you want to go view the commits to that tree, please feel free. It doesn’t take much to see that we commit to that tree all day, and every time we change something, the documentation gets rebuilt. How frequently? Well, on a typical day we will generate 10-15 new versions of each reference manual. It’s actually difficult to rebuild more frequently than that due to the sheer size of the documentation.

If you want to check the build date of the documentation, check the intro/preface of each document. The build and build date information is included there.

Myth

The MySQL documentation is small and unused.

Reality

You’d be amazed how many people need to be told RTFM, but a surprising number of people who criticize the MySQL documentation have actually never read it, or, they looked at it years ago and haven’t bothered to look recently because they couldn’t find what they were looking for before.

The reality is that our documentation is over 2000 pages per reference manual, which means over 10,000 pages now just for MySQL. There are hundreds more pages on the GUI tools, Workbench, Cluster/NDBAPI, and the Enterprise Monitor.

As to the popularity, the hits to the online pages of our manual exceed the hits to every other section of the MySQL website by a significant factor. For the downloadable formats we get an average of 200,000 downloads in all the various formats each month, with occasional spikes up to 800,000. For the online manuals, the documentation pages make up about 45% of all the traffic on mysql.com. Or to say it another way, we account for almost half of all the web traffic that MySQL receives, including downloads.

In short, we have no shortage of interested readers.

Myth

Docs team don’t read comments

Reality

Actually, we all get an email each time you post a comment and all of us will read it, determine whether it is suitable, useful, or (occasionally) spam, and either ignore it, delete it, or comment on it accordingly. Often that will happen within minutes of your leaving the comment. If there’s a non-standard reply to your comment, you’ll get that too.

Now, we are aware that the comments system has it’s faults. For one, we have one comment system for all the different versions of the manual, which means comments can be confusing and even misleading. We’re fixing that. We’re also trying to address the problem that some comments are really tips, while others are just plain comments and observations.

Any other comments or criticisms, let us know. We may not be unaware of the problem, but if we know your pain we can do something about it.

Myth

Docs team don’t accept bugs or corrections

Reality

You can report a bug or correction to us using the standard bugs.mysql.com, or drop us an email to docs@mysql.com.

Myth

Docs are ‘closed source’

Reality

The docs are not closed source – you can download the DocBook XML and the files and tools required to build them (well, beyond the XML parsers, Perl, and other bits and pieces). You can get hold of the repos (via SVN), on the Tech Resources page.

That said, we don’t allow anybody to commit changes, but see the response above for information on how to provide changes and fixes. Again this is something we are working to improve on.

Myth

MySQL Documentation is not distributable

Reality

This mostly comes out of the fuss around Debian dropping the man pages from their MySQL distributions You can see the description why here: MySQL Documentation and Debian/Ubuntu.

The short answer is that it is a mis-understanding in our license for the documentation, which is not released under the same license as MySQL. You can provide documentation if you provide MySQL, but not on it’s own. The reason for this is that our documentation is updated so regularly that we want to ensure that we only get genuine, up to date, versions of our documentation out there. Trust me, do a search for MySQL and some term and you will find versions of the manual that are months, or even years out of date, which is no help to anybody.

It’s about trying to make our documentation readable and usable and not misleading.

OK that’s enough myths busted today, but if you hear any more, or just have additional questions, feel free to ask.

The material is part of the Curriculum Development Resources at OpenSolaris.org and is available as PDFs for download. The documents are short and easy to read, but packed with lots of useful information and a good read for anybody interested in understanding more about the technology and functionality in OpenSolaris.

So who is he? Well, Stefan is the guy that keeps the rest of us in the docs team in check and makes sure we do what we’re asked, when we’re are asked and that all of the machinery, legalities and management tasks happen in the background. Without him we really couldn’t function as effectively as we do.

It’s wonderful to see some other Docs team members getting in on the act (to be fair to the rest of the team, Jon is also a blogger). We are all writers, you would think the blogging would come as a natural extension.

Behind the tease is the simple fact that the improved system for MySQL University I was talking about is getting a trial run this week.

We’ve been trying out Dimdim for web conferencing with David van Couvering and I have to say I’m pretty impressed.

We chatted, we tried out screen-sharing, presentations and the whiteboard functionality and it all worked really nicely.

We’re going to be using it for the MySQL University session this week, Using MySQL with NetBeans. Space will be limited, but feel free to join us if you can.

On Windows, you dont have valgrind, but Microsoft do provide a free native debugging tool, called the user-mode dump heap (UMDH) tool. This performs a similar function to valgrind to determine memory leaks.

Vladislav Vaintroub, who works on the Falcon team and is one of our resident Windows experts provides the following how-to for using UMDH:

1. Download and install debugging tools for Windows from here
   MS Debugging Tools
   Install 64 bit version if you’re on 64 bit Windows and 32 bit version
   otherwise.

2. Change the PATH environment variable to include bin directory of Debugging tools.
   On my system, I added
   C:\Program Files\Debugging Tools for Windows 64-bit to the PATH.

3. Instruct OS to collect allocation stack for mysqld with gflags -i
mysqld.exe +ust.
   On Vista and later, this should be done in “elevated” command prompt,
   it requires admin privileges.

   Now collect the leak information. The mode of operation is that: take the
   heap snapshot once, and after some load take it once again. Compare
   snapshots and output leak info.

4. Preparation : setup debug symbol path.
   In the command prompt window, do

   set _NT_SYMBOL_PATH= srv*C:\websymbols*http://msdl.microsoft.com/download/symbols;G:\bzr\mysql-6.0\sql\Debug

   Adjust second path component for your needs, it should include directory
   where mysqld.exe is.

5. Start mysqld and run it for some minutes

6. Take first heap snapshot

   umdh -p:6768 -f:dump1

   Where -p: actually, PID of my mysqld was 6768.

7. Let mysqld run for another some minutes

8. Take second heap snapshot

   umdh -p:6768 -f:dump2

9. Compare snapshots

   umdh -v dump1 dump2 > dump.compare.txt

10. Examine the result output file. It is human readable, but all numbers are
    in hex, to scare everyone except geeks.

11. gflags -i mysqld.exe -ust

    Instruct OS not to collect mysqld user mode stacks for allocations
    anymore.

These are 10 steps and it sounds like much work, but in reality it takes 15
minutes first time you do it and 5 minutes next time.

Additional information is given in Microsoft KB article about UMDH
KB 268343.

Van Lindberg introduces the book by talking about the comments that end up on Slashdot.org, almost certainly prefixed by the expression IANAL (I Am Not A Lawyer) where people defend, discuss, and rip people up about the legalities of open source and the various licenses. Van Lindberg also talks about how he spends much of his time translating the contents of various legal documents into engineer speak and back again.

Despite being a proponent and long time user of free software and open source for the best part of my working life, I’ll admit to being completely ignorant of many of the issues. This isn’t through lack of interest, but I’d rather leave those discussions and decisions to people who know, and it’s clear that Van Lindberg not only knows the subject, but he also knows how to make it interesting to those of us who actually have to work within the confines of rules and regulations.

I’m still reading and learning a lot of the ins and outs of copyright, company agreements, and individual licenses and details. There’s a lot of material and detail included here.

I’ll have a full review when I’ve finished. Until then, if you have even a passing interest in the various licensing, legal and IP issues with open source, check out the book for a proper read.

Personally, I think they’re great. Obviously many times I am scribe and am there for the sessions, but I listen to lots of the sessions anyway, and I’m yet to be disappointed by the content. What’s really great is that in all the cases the person you are listening to is probably the person that either developed, or helped drive development of the particular function, or, in the case of some of the external tools (EC2, for example), these guys are expert in it. The experience is not quite as thrilling as attending the MySQL User Conference, but the content is just the same.

The problem is that despite all the work we do to get the presenters, interesting topics, and promotion of the upcoming sessions, we don’t always get as many attendees as we want or expect.

So, I’m wondering why this should be the case. We know that the current presentation system is not ideal (and we’re working on that), but I’m interested to hear people’s opinions on MySQL University. If you want to help shape the future of MySQL University, then comment here, and either answer the questions below, or make up your own.

• Have you attended any MySQL University sessions. How many?
• How would you rate the sessions generally? A simple good or bad will do
• If you haven’t attended any sessions, or don’t regularly attend them, why not?
• Have you ever looked at/listened to the past sessions that provide on MySQL Forge?

Please, I’m interested to hear.

Interestingly I’ve just found this item in my blog post queue from way back in 2006, Jonathan Schwartz: The Rise of the General Purpose System, where Jonathan pretty much spells out that general purpose was the way to go. Obviously the X4150 is an embodiment of that belief.

I have had the good fortune of working with the MySQL Enterprise Monitor team as an advisor, and more recently in writing the documentation, for the last 18 months. We’ve had some great fun at meetings in Amsterdam, Heidelberg, Santa Cruz and recently Riga. In the meeting rooms we are professional, but fun. But in the evenings we’ve gone out and just had plain good fun. Unless you’ve experienced a full day, or even week, of non-stop meetings for 12 hours a day you have no idea how important it is to kick back in the evenings. That, as a team, we are still able to have a good time at the end of each day while still being in the same room together is a good indication of how well we all get on.

Why am I telling you this? Because there is an opportunity to come and work for MySQL Enterprise Monitor team as a GUI developer, and if you are going to join us, you need to be as much fun at the dinner table as you are in the meeting room.

We are looking for talented people, obviously, and you are going to need to be good both with the design and the programming aspects of the GUi development. We use frameworks like Hibernate and Spring, and build our interfaces using DHTML and AJAX. If you know about MySQL and scale-up/out environments, that would be even better.

If it sounds like a good fit for you, and you just happen to want to have a good time to boot, apply here.

What they don’t tell you on the official pages is how much you will be expected to also get on with the rest of the team. It’s really important, because when we have those meetings we’ll be looking to the new guy to provide us with entertainment and interesting stories, at least until the next guy comes along.

I’m just doing a piece on XML schemas and would love to be able to create a ready to run sample of SOAP and WSDL XML files without having to actually create the example by hand. Just a quick ‘give me this function, these arguments’ and have it generate it.

Years I go I thought about this for things like contacts and book projects.

This morning, we learned what it meant to be a pirate in terms of patents, copyright and now politics with the Pirate Party. Don’t let the scary name put you off – these guys are about making all of us consumers (of software, video, audio, books, etc.) more in control of information. Please support these guys by visiting Piratpartiet.se.

Next we had the Scalability Panel with representatives from Facebook, Fotolog, Sun, YouTube, Flickr and Wikipedia talking about the problems (and some of the solutions) they have taken to approach the bane of any web company.

One option for scalability of course is just not to cover the problem yourself. Skip the issues of server rooms and use Amazon’s various services (EC2, S3, etc). If you need convincing, try Werner Vogels’ talk.

Yesterday, in a short but sweet visit to the podium, Rich Green talked about MySQL and Sun and it will all work. If you have concerns about the integration, this is a good place to get the situation from the guy who will make sure it doesn’t go wrong. And of course Rich will answer to Jonathan.

Marten talked about the significance of Storage engines and the significance of your data.

For a full list, try Zack’s profile page.

• MySQL Conference Keynote 2008 – Marten Mickos
• MySQL Conference Keynote 2008 – Jonathan Schwartz
• MySQL Conference Keynote 2008 – Werner Vogels

Sadly these are only snippets, but if you like what you see, make sure to book your place for next year’s conference early!

The new version comes with a number of new features, some of which I’ll be detaileding over the coming weeks, but as a headline:

• New layout and structure
• New Inspire Me section for quick access to selected recipes
• Your own personal Kitchen area where you can manage your preferences, your cookbooks, recipes, shopping list, and connect with other users.
• Add your own recipes – you can now add a recipe, have the nutrition automatically calculated, and then share that recipe with others.
• RSS feeds for your cookbook, recipes, and Inspire me and automated searches
• Improved searching and nutrition information

And there’s more to come – please start using the site and let us know what you think, or if you find any problems.

The service is an interface RSS reader – you get to choose the topic and the feed (currently only four static feeds are provided), then it will read out the feed content.

You can try out the demo by calling:

• Skype: +99000936 9991260725
• US (freephone): (800) 289-5570, then using PIN 9991260725

Occasionally the hosting times out, in which case, please contact me and I’ll check it out and restart or reboot the service.