Skip to content
Jonathan Schnee edited this page Jul 14, 2021 · 10 revisions

Picking

📥

Tutorial 3 Picking

A common task in realtime 3D applications is to find out which objects in the 3D scene are located below a certain 2D pixel position on the screen, e.g. where a user has just clicked with the mouse or performed a touch gesture with his finger. FUSEE provides the class ScenePicker which can be used to accomplish this task.

Like the SceneRendererForward and also the described FindNodes() method, picking performs a traversal of the scene graph, i.e. all nodes and all necessary components are visited. While in rendering the visit results in each component making its contribution to the image to be rendered and in searching a search criterion is checked during the visit, in picking - similar to rendering - the vertices of each triangle of the geometry are converted into screen coordinates so that a point-in-triangle test can then be performed.

Whenever this test is positive (point is in the triangle), a set of information is collected that can then be evaluated by the user. This information includes:

  • The straight traversed node
  • The straight traversed (mesh) component
  • The index of the first point of the triangle in the 'triangles' list, for which the point-in-triangle test was positive
  • The so-called barycentric coordinates, which indicate where exactly within the triangle the point lies
  • Model, view and projection matrix, with which the transformation of the model coordinates in screen coordinates.

This information is stored in the class PickResultsummarized.

With this information not only the 3D objects lying under a screen pixel can be identified, these can also be sorted along the z-coordinate, so that e.g. the object lying furthest in front can be found out. In addition, the triangle that was hit can also be identified, as well as the exact position of the "point of impact" can be calculated, in model world or screen coordinates.

👨‍🔧 TODO

  • Add the following three fields ("class variables") to the app class:
private ScenePicker _scenePicker
private PickResult _currentPick;
private float4 _oldColor;

and inserts the following code in the Init() method after loading the scene:

_scenePicker = new ScenePicker(_scene);
  • Inserts the following code into the RenderAFrame() method AFTER the camera has been set .
// Setup the camera 
RC.View = float4x4.CreateTranslation(0, 0, 40) * float4x4.CreateRotationX(->(float) Math.Atan(15.0 / 40.0));
if (Mouse.LeftButton)
{
  float2 pickPosClip = Mouse.Position * new float2(2.0f / Width, -2.0f / Height) + new float2(-1, 1);

  PickResult newPick = _scenePicker.Pick(RC, pickPosClip).OrderBy(pr => pr.ClipPos.z).FirstOrDefault();

  if (newPick?.Node != _currentPick?.Node)
  {
    if (_currentPick != null)
    {
      var ef = _currentPick.Node.GetComponent<DefaultSurfaceEffect>();
      ef.SurfaceInput.Albedo = _oldColor;
    }
    if (newPick != null)
    {
      var ef = newPick.Node.GetComponent<SurfaceEffect>();
      _oldColor = ef.SurfaceInput.Albedo;
      ef.SurfaceInput.Albedo = (float4) ColorUint.OrangeRed;
    }
    _currentPick = newPick;
  }
}
  • First, the current mouse position, which contains pixel coordinates, is converted into so-called 2D clip coordinates. These have the origin in the center of the output window and at the window border respectively 1 (right and top), and -1 (left and bottom).
  • The call to _scenePicker.Pick() performs the traversal and returns an unsorted list > of pick results. of pick results.
  • In doing so, Pick() is passed the RenderContext as a parameter, so that it can calculate valid screen coordinates from the model coordinates.
  • If the list is not empty, it will be sorted (OrderBy()) according to the z screen coordinate of the pick events (pr => pr.ClipPos.z). Smaller z values come to the front. The pick event with the smallest z value is the one closest to the viewer.
  • The first pick event if any (FirstOrDefault()) is stored in the newPick variable.
  • The name of the node associated to the pick event is displayed on the screen. objects of your 3D scene. The clicked single parts should be marked by the highlight color (1, 0.4f, 0.4f).

Congratulations you have completed the tutorial. Here you can find the finished application with all three functions Tutorial Finished

Clone this wiki locally