Month: February 2011

Ever since we took over the PowerOffice tools in December of 2008 there has been a bug in the PowerToolbar that I had just about given up on ever finding. It was to a point that I have been considering pulling the product off the market if I couldn't find and fix this bug, because this was becoming more and more of an issue.

The problem would only manifest on Vista, Windows Server 2008 and Windows 7 machines. The code that themes the menus is different for XP and older operating systems and there is no problem. What happens in Vista and newer operating systems is that some menus were not themed at all and this would offset the settings so that icons would be out of sequence, some items wouldn't be themed correctly, menu items would disappear, text was missing etc.

Last week I decided to take this week and dig into this and either find it and fix it or basically give up on it and throw in the towel. This was the fourth time I dedicated a week or so of work to try to find this. I couldn't get going on this for real until Wednesday and spent Wednesday and Thursday tracking things. Yesterday, Friday, I had isolated the problem to what appeared to be the WM_DRAWITEM message not firing for some of the menu items and by last night I was completely confused. It made no sense that messages that fire when a menu item is drawn were not firing. No sense at all! So I decided to do the sensible thing and sleep on it!

When I got up this morning I had formed a bit of a battle plan and decided to start from the other end of things. Write code to track down the handles of the menus and then duplicate the steps that PowerToolbar takes in order to gather information about the menus and see if something went missing. My thought was that perhaps there was something in the menus that caused them not to be traversed by the API calls made to gather information about the menus, items and sub-menus.

I wrote a simple class that looped through the controls on the appframe window and put the menus and menu items into a queue. I then used GetMenuItemCount() to get the number of items in each menu as well as the handle (PROP:Handle) for each menu. I used OutputDebugString, my favorite method of debugging, to output the information into DebugView and then checked out each menu and the information. I started noticing a trend! Every menu that was not themed correctly had a negative value for the handle. I was on to something, but what?

After a lot of experimenting it was obvious that something was preventing the code from detecting the menus with negative handles and thus not theming them, setting everything off. I checked a couple of methods that gather information about the menus and menu items and also set the menu to be owner drawn. It didn't take long for one particular API to stand out: GetSubMenu(). It retrieves the handle of sub-menu from a menu handle and thus makes it possible to "walk" the menu structure using API calls.

But it wasn't the actual call to GetSubMenu() that was interesting, rather the check that came after it:

hSubMenu = PTB:GetSubMenu(hMenu, Pos-1)
If hSubMenu > 0 Then SELF.GetMenuInfo(hSubMenu, Feq).

Looks good to me! But when I carefully read the information about GetMenuInfo on MSDN I realized that the function returns 0 ONLY if it fails or if the handle is not returned. The handle can be negative and now everything came together in my head and I realized this was why it couldn't handle the negative handles - it would only recurse if the menu handle was a positive number greater than zero!.

The bottom line was that I had to add the "<" character in two places and the problem was solved. I changed the code above to:

hSubMenu = PTB:GetSubMenu(hMenu, Pos-1)
If hSubMenu <> 0 Then SELF.GetMenuInfo(hSubMenu, Feq).

in two places and the problem disappeared! This is already out for beta testing and I hope to release this early next week! I feel very good about finally finding this and figuring out the fix.