Where is the caret in a text selection?

This wasn't intended but eventually becomes a followup on "Where is the caret?"

There are two directions to select some text in a rich edit: from left to right, or from right to left. The initial position of a selection is called an anchor point, and the ending position is called an active end. These are the names given in the EM_SETSEL documentation. The four other related messages and notification are: EM_GETSEL, EM_EXSETSEL, EM_EXSETSEL, and EN_SELCHANGE.

Since the caret marks the active position, to reshow or do something interesting with the caret, I need to know where the anchor and active positions are, given a selection range. As it turns out, all of above messages or notification (except EM_SETSEL) assume the start/min being anchor, and end/max being active.

Quite surprisingly, TOM can provide this information, through ITextDocument:GetSelection and ITextSelection:GetFlags (tomSelStartActive). I said surprising because TOM looks more like wrapper interfaces around Word and rich edit controls, but hey, what do I know?

How to pick folder with OpenFileDialog?

OpenFileDialog allows selection of files, and FolderBrowserDialog allows selection of folders. For various legitimate reasons, I prefer to use OpenFileDialog for folders instead. The question is, how?

Overriding FileDialog.HookProc to obtain the dialog handle to change the dialog behavior is my first attempt, but it is immediately obvious that OpenFileDialog is sealed and cannot be inherited, and I don't feel like reimplementing everything with GetOpenFileName. So an alternative way to get the dialog handle is to cook up a dummy window (as in FileDialog > DummyOwner > RealOwner) and watch for WM_ACTIVATE in the overriden WndProc. As always, be extra careful when playing with window activation.

Now, with the dialog handle, and later the child notification dialog handle, I can do pretty much what I want. I can hide and rename various controls on the dialog, handle BN_CLICKED to accept folder, and more.

A few gotchas include having to do manual marshaling using COM task memory, but no big deal.

Next up is the Vista version.

What's up with the Server 2008 desktop graphics bugs?

When Desktop Experience is installed, Server 2008 provides similar visuals as Vista. One thing I noticed though, is that the desktop doesn't act the same as in Vista:

1. The drop shadow on icon text may disappear on hover.
2. The selection rectangle has dotted outline as in Windows XP* instead of translucent.
3. The selected icons don't have translucent background.

It turns out that the desktop SysListView32 control is styled differently. So now I have a simple startup program that finds the Progman window, then the SHELLDLL_DefView window, then the SysListView32 window, and adds the missing extended styles LVS_EX_DOUBLEBUFFER, LVS_EX_TRANSPARENTBKGND, and LVS_EX_TRANSPARENTSHADOWTEXT.

A more robust solution would create a message-only window in the background, listen for the TaskbarCreated message, and reapply the styles to the recreated SysListView32.

These would take care of 1 and 2, but not 3. May be a call to SetWindowTheme would work, but I haven't tried.

* Similar techniques can be used on XP to get the translucent selection rectangle, but will clear the wallpaper.

Why doesn't WTSConnectSession work when I first tried it?

Because the MSDN documentation on WTSConnectSession is wrong.

If you follow it:

Parameters

LogonId [in]

The logon ID of the current session, where the user of that session has permissions to connect to an existing session.

TargetLogonId [in]

The logon ID of the session that you want to connect to.

And call the function this way, your program will merely disconnect your current session with ERROR_NOT_CONNECTED, leaving you to wonder if the execution of the program stops in the middle of the call.

Once you find out that WTSConnectSession is just a wrapper for the undocumented WinStationConnect, which exists in earlier versions of Windows than its wrapper, you should realize that the first two parameters as documented are swapped.

What's the scope of window handles?

Someone stated (though I've yet to find any supporting MSDN documents) that window handles are global to all processes belonging to the same window station. Yet, if you somehow obtain a handle to a window belonging to another desktop, and try to send messages to it, SendMessage may fail with ERROR_INVALID_WINDOW_HANDLE. Why?

If window handles are truly global, then, say, thread 'a' on desktop 'A' should be able to use handles from thread 'b' on desktop 'B', assuming that both desktops belong to the same window station, right? It seems that some magic needs to happen for the handle table to be visible to other desktops, that is, thread 'a' needs to obtain a handle to desktop 'B' with read access.

It appears logical, but needs to be clearly documented!

Where is the caret?

Carefully minded will spot at least one subtle difference between the Visual Studio editor, and the RichEdit control: the caret in selection. Why oh why I can get this cute blinking cursor when I select some text in Visual Studio, but not in WordPad or in RichTextBox?

Maybe it's hidden for some bizarre reasons, I'll just call ShowCaret to bring it back. Argh, doesn't work? OK, may be HideCaret is called multiple times, so I'll just call ShowCaret a zillion times to see what gives. What? The caret is actually destroyed, and I have to call CreateCaret?

How am I supposed to know the height of the caret? It's not exactly Font.Height, and I can't always get the position of two lines and take the difference.

Text Object Model (TOM) comes into rescue. It's implemented by RichEdit (not Edit, so no hope there, yet). Get an IRichEditOle with EM_GETOLEINTERFACE, then an ITextDocument, then an ITextRange from ITextDocument::Range, and finally the coordinates from ITextRange::GetPoint.

I'll leave it as an exercise to declare the TOM interfaces for .net. Yea this whole thing is purely artistic...

MSDN Documentation. Follow or Not Follow?

When reading MSDN, often time there are sentences like "do not do XXX" or "you have to do YYY" without further explanation. This time it is on the OFNHookProc page:

Do not call the EndDialog function from the hook procedure.

It happened that I also came across the HookProc implementation in FileDialog, using the .Net Reflector:

[SecurityPermission(SecurityAction.LinkDemand, Flags = SecurityPermissionFlag.UnmanagedCode)]
protected override IntPtr HookProc(IntPtr hWnd, int msg, IntPtr wparam, IntPtr lparam) {
    // ......
    } catch {
        if (this.dialogHWnd != IntPtr.Zero) {
            UnsafeNativeMethods.EndDialog(new HandleRef(this, this.dialogHWnd), IntPtr.Zero);
        }
        throw;
    }
    // ......
}

May be the .net team know win32 in and out that outsider rules don't apply to them; or may be sometimes it is acceptable to ignore MSDN.

RichTextBox is not a simple RichEdit wrapper?

There are several annoyances with the Rich Edit control. For example, using EM_STREAMIN / EM_STREAMOUT with EditStreamCallback will clear its undo and redo queues. Why would this caused by simply reading the stream is beyond me.

The .net RichTextBox inherits this problem, which means accessing its Text, Rtf, and Lines properties will clear the queues. What's more, calling its GetCharIndexFromPosition and GetPositionFromCharIndex functions will also clear the queues. Why is that? Don't GetCharIndexFromPosition and GetPositionFromCharIndex just wrap EM_CHARFROMPOS and EM_POSFROMCHAR?

Workarounds?

• Replace functions GetCharIndexFromPosition / GetPositionFromCharIndex with EM_CHARFROMPOS / EM_POSFROMCHAR
• Replace property Text with EM_GETTEXTEX / RichTextBox.SelectAll / SelectedText
• Replace property Lines with EM_GETTEXTEX / String.Split / Join
• Replace property Rtf with... no idea.

Why ToolStrip responses to WM_MOUSEACTIVATE differently?

When a user clicks on a .Net ToolStrip (or MsoCommandBar) on which the top-level window is not focused, the window will be focused (duh) but the ToolStrip will not process the associated mouse event. For example, when a user clicks on the New button when Microsoft Outlook is not focused, the new message dialog will not appear.

This is apparently due to the ToolStrip returning MA_ACTIVATEANDEAT on WM_MOUSEACTIVATE. I found this when I was coding my ToolStrip based tab control. A simple fix would be to return MA_ACTIVATE instead:

protected const int WM_MOUSEACTIVATE = 0x0021;
protected const int MA_ACTIVATE = 1;
protected override void WndProc(ref Message m) {
   if (m.Msg == WM_MOUSEACTIVATE) {
       m.Result = new IntPtr(MA_ACTIVATE);
       return;
   }
   base.WndProc(ref m);
}

Annoying, and won't be cross platform, but at least it works!

What's wrong with WS_EX_LAYERED and WS_EX_COMPOSITED?

From MSDN:

WS_EX_COMPOSITED

Windows XP: Paints all descendants of a window in bottom-to-top painting order using double-buffering. For more information, see Remarks. This cannot be used if the window has a class style of either CS_OWNDC or CS_CLASSDC.

WS_EX_LAYERED

Windows 2000/XP: Creates a layered window. Note that this cannot be used for child windows. Also, this cannot be used if the window has a class style of either CS_OWNDC or CS_CLASSDC.

Remarks

Windows XP: With WS_EX_COMPOSITED set, all descendants of a window get bottom-to-top painting order using double-buffering. Bottom-to-top painting order allows a descendent window to have translucency (alpha) and transparency (color-key) effects, but only if the descendent window also has the WS_EX_TRANSPARENT bit set. Double-buffering allows the window and its descendents to be painted without flicker.

If WS_EX_COMPOSITED is set on a dialog window (e.g. to reduce flickers from child windows resize), then, with the default XP theme, mouse hover on the minimize button or mouse down on the close button will have no visual feedback. It seems that the back buffer for the non client area does not get updated. One obvious workaround is to only add WS_EX_COMPOSITED on WM_ENTERSIZEMOVE and remove it on WM_EXITSIZEMOVE.

What if WS_EX_LAYERED is also set? The above workaround will not work, and worse, the client area will have all sorts of update problems: edit window will not show or receive blinking insertion point, push button will not show sunken/raised border on mouse down/up. Again it seems that the back buffer is not updated. Yet another obvious workaround is to only add WS_EX_LAYERED on WM_ENTERSIZEMOVE and remove it on WM_EXITSIZEMOVE.

What's next? I'm waiting to be hit by more poorly documented styling bugs.