Augmenting 3D graphics with 2D labels

[Jon Harrop]

I am writing a 3D graphing component built upon WPF and would like to have
2D vector graphics (e.g. typeset mathematics) as labels laid out from 3D
coordinates. For example, a tick on an axis has a 3D coordinate and its 2D
label might be right-aligned to the 2D projection of that 3D point.

The following Mathematica plot illustrates the functionality I am after:

http://math.arizona.edu/~goriely/M32...omplexFunc.jpg

I have done some Googling but not found a solution. Some people use 3D
labels but these are ugly and suffer from various artefacts (e.g. they
become heavily distorted or disappear completely when viewed at the wrong
angle).

The best solution I can imagine is a 2D DrawingVisual overlaid over my 3D
Viewport but I need a function to project 3D coordinates onto 2D
coordinates and WPF does not seem to provide one. I can reverse engineer
the projection algorithm used by WPF but Microsoft may change the algorithm
in the future and break my code.

Any ideas?

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Jon Harrop]

Jon Harrop wrote:

I am writing a 3D graphing component built upon WPF and would like to have
2D vector graphics (e.g. typeset mathematics) as labels laid out from 3D
coordinates. For example, a tick on an axis has a 3D coordinate and its 2D
label might be right-aligned to the 2D projection of that 3D point.

The following Mathematica plot illustrates the functionality I am after:

http://math.arizona.edu/~goriely/M32...omplexFunc.jpg

I have done some Googling but not found a solution. Some people use 3D
labels but these are ugly and suffer from various artefacts (e.g. they
become heavily distorted or disappear completely when viewed at the wrong
angle).

The best solution I can imagine is a 2D DrawingVisual overlaid over my 3D
Viewport but I need a function to project 3D coordinates onto 2D
coordinates and WPF does not seem to provide one. I can reverse engineer
the projection algorithm used by WPF but Microsoft may change the
algorithm in the future and break my code.

Any ideas?

To answer my own question: I finally got this working reliably after several
days of work and it is not easy. Moreover, there is a lot of unfortunate
ugliness because WPF exposes less functionality than alternatives like
OpenGL so you have to do a lot of work just to get something that works but
is inefficient.

To overlay 2D graphics over 3D graphics, I initially tried StackPanel,
DockPanel and the Canvas. StackPanel and DockPanel are not suitable because
their layout algorithms either overlay but provide too much space (the
default infinite space crashes WPF if it reaches a Viewport3D) or they
stack or dock the Viewports as expected but not as wanted. Canvas lets you
place the controls but it also provides infinite space for layout and that
crashes WPF. You can make the space finite but it needs to be a function of
the space available to the parent and that is not available.

So the most elegant solution I came up with was to derive from Panel and
implement my own MeasureOverride and ArrangeOverride, which were derived
from those in the other controls, gleaned from PresentationCore.dll using
Reflector.

That works beautifully in providing a 3D projection with a 2D overlay but
there is still the problem of projecting 3D coordinates into 2D
coordinates. Again, the required code can all be seen in
PresentationCore.dll using reflector but Microsoft have made it internal so
it cannot be reused. So the only solution is to reimplement this yourself.
I copied their code for ViewMatrix and ProjectionMatrix for a
PerspectiveCamera and used it to create a MatrixCamera from explicit
matrices, checked that it gave the same results and then went on. Next, I
used the uncommon Point4D type to transform the 3D coordinate without
losing the essential W coordinate of the resulting point. The projected 2D
coordinate is then (x/w, y/w) as usual.

I am extremely happy with the results (most notably that it should work
reliably on customers' computers) but a bit miffed that it was so much
harder to implement with WPF than OpenGL.

The next problem I face is drawing lines in a 3D plot. Apparently WPF also
lacks this functionality so even though I have hardware capable of
accelerating anti-aliased line rendering with z-buffering (fully accessible
from OpenGL), WPF is going to force me to implement this in software by
retesselating everything into 3D triangles every time the view or scene
changes. Ugh.

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[=?Utf-8?B?SmFrb2IgQ2hyaXN0ZW5zZW4=?=]

I have done something similar using a somewhat different approach. I had a
canvas containing the Viewport3D. In the 3D scene I placed 2D visuals using
Viewport2DVisual3D. To find the coordinates of these visuals relative to the
canvas, I used Visual.PointToScreen. I could then map these coordinates to
the canvas using Canvas.PointFromScreen and place labels etc. on the canvas
using the coordinates.

It is a very simple approach and not as general as your method of
calculating the projection as it requires visuals in the 3D scene but it is
very simple and easy to use.

BTW, using tooltips of visuals in the 3D scene is another extremely easy way
of showing content in response to hovering the mouse.

Kind regards, Jakob.

"Jon Harrop" wrote:

Jon Harrop wrote:

I am writing a 3D graphing component built upon WPF and would like to have
2D vector graphics (e.g. typeset mathematics) as labels laid out from 3D
coordinates. For example, a tick on an axis has a 3D coordinate and its 2D
label might be right-aligned to the 2D projection of that 3D point.

The following Mathematica plot illustrates the functionality I am after:

http://math.arizona.edu/~goriely/M32...omplexFunc.jpg

I have done some Googling but not found a solution. Some people use 3D
labels but these are ugly and suffer from various artefacts (e.g. they
become heavily distorted or disappear completely when viewed at the wrong
angle).

The best solution I can imagine is a 2D DrawingVisual overlaid over my 3D
Viewport but I need a function to project 3D coordinates onto 2D
coordinates and WPF does not seem to provide one. I can reverse engineer
the projection algorithm used by WPF but Microsoft may change the
algorithm in the future and break my code.

Any ideas?

To answer my own question: I finally got this working reliably after several
days of work and it is not easy. Moreover, there is a lot of unfortunate
ugliness because WPF exposes less functionality than alternatives like
OpenGL so you have to do a lot of work just to get something that works but
is inefficient.

To overlay 2D graphics over 3D graphics, I initially tried StackPanel,
DockPanel and the Canvas. StackPanel and DockPanel are not suitable because
their layout algorithms either overlay but provide too much space (the
default infinite space crashes WPF if it reaches a Viewport3D) or they
stack or dock the Viewports as expected but not as wanted. Canvas lets you
place the controls but it also provides infinite space for layout and that
crashes WPF. You can make the space finite but it needs to be a function of
the space available to the parent and that is not available.

So the most elegant solution I came up with was to derive from Panel and
implement my own MeasureOverride and ArrangeOverride, which were derived
from those in the other controls, gleaned from PresentationCore.dll using
Reflector.

That works beautifully in providing a 3D projection with a 2D overlay but
there is still the problem of projecting 3D coordinates into 2D
coordinates. Again, the required code can all be seen in
PresentationCore.dll using reflector but Microsoft have made it internal so
it cannot be reused. So the only solution is to reimplement this yourself.
I copied their code for ViewMatrix and ProjectionMatrix for a
PerspectiveCamera and used it to create a MatrixCamera from explicit
matrices, checked that it gave the same results and then went on. Next, I
used the uncommon Point4D type to transform the 3D coordinate without
losing the essential W coordinate of the resulting point. The projected 2D
coordinate is then (x/w, y/w) as usual.

I am extremely happy with the results (most notably that it should work
reliably on customers' computers) but a bit miffed that it was so much
harder to implement with WPF than OpenGL.

The next problem I face is drawing lines in a 3D plot. Apparently WPF also
lacks this functionality so even though I have hardware capable of
accelerating anti-aliased line rendering with z-buffering (fully accessible
from OpenGL), WPF is going to force me to implement this in software by
retesselating everything into 3D triangles every time the view or scene
changes. Ugh.

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[Jon Harrop]

Jakob Christensen wrote:

I have done something similar using a somewhat different approach. I had
a canvas containing the Viewport3D. In the 3D scene I placed 2D visuals
using Viewport2DVisual3D. To find the coordinates of these visuals
relative to the canvas, I used Visual.PointToScreen. I could then map
these coordinates to the canvas using Canvas.PointFromScreen and place
labels etc. on the canvas using the coordinates.

Very interesting. That is probably what Microsoft had intended for me to
use. I shall check that out. Thanks!

It is a very simple approach and not as general as your method of
calculating the projection as it requires visuals in the 3D scene but it
is very simple and easy to use.

You say that it is not as general but what can I do with my technique that
you cannot do with yours? Are your 2D graphics antialiased? Are they
z-buffered so that 3D objects can obscure them?

Also, I am now considering writing my own implementation of 2D lines
projected back into 3D world space and rendered as triangles. Given the
number of existing libraries that already implement this functionality I
assume there is no easy solution to plotting 3D lines using WPF?

BTW, using tooltips of visuals in the 3D scene is another extremely easy
way of showing content in response to hovering the mouse.

For some reason I didn't cite our product pages after we shipped the code
but they clearly show the effects I was after (the ticks):

http://www.ffconsultancy.com/product...ion/demo1.html
http://www.ffconsultancy.com/product...ion/demo2.html

For an update: I am really happy with the resulting 2D/3D visual effect for
real-time interactive visualizations integrated with Visual Studio. The
software will now be easy to maintain and customers are very happy with
reliability. Performance is adequate on a fast machine but vastly slower
than OpenGL under Linux on the same machine, which is a bit of a shame.

--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u

[=?Utf-8?B?SmFrb2IgQ2hyaXN0ZW5zZW4=?=]

You say that it is not as general but what can I do with my technique that

you cannot do with yours? Are your 2D graphics antialiased? Are they
z-buffered so that 3D objects can obscure them?

My method requires a Viewport2DVisual3D containing 2D visuals. The visuals
can be invisible though.

>
Also, I am now considering writing my own implementation of 2D lines
projected back into 3D world space and rendered as triangles. Given the
number of existing libraries that already implement this functionality I
assume there is no easy solution to plotting 3D lines using WPF?

No, there is no easy solution to plotting lines.

http://www.ffconsultancy.com/product...ion/demo1.html
http://www.ffconsultancy.com/product...ion/demo2.html

Looks nice!