Download Demo Files ZIP 35KB

TouchDesigner*, created by Derivative, is a popular platform/program used worldwide for interactivity and real-time animations during live performances as well as rendering 3D animation sequences, building mapping, installations and recently, VR work. The support of the Intel® RealSense™ camera in TouchDesigner makes it an even more versatile and powerful tool. Also useful is the ability to import objects and animations into TouchDesigner from other 3D packages using .fbx files, as well as taking in rendered animations and images.

In this two-part article I explain how the Intel RealSense camera is integrated into and can be used in TouchDesigner. The demos in Part 1 use the Intel RealSense camera TOP node. The demos in Part 2 use the CHOP node. In Part 2, I also explain how to create VR and full-dome sequences in combination with the Intel RealSense camera. I show how TouchDesigner’s Oculus Rift node can be used in conjunction with the Intel RealSense camera. Both Part 1 and 2 include animations and downloadable TouchDesigner files, .toe files, which can be used to follow along. To get the TouchDesigner (.toe) files click on the button on the top of the article. In addition, a free noncommercial copy of TouchDesigner which is fully functional (except that the highest resolution has been limited to 1280 by 1280), is available.

Note: There are currently two types of Intel RealSense cameras, the short range F200, and the longer-range R200. The R200 with its tiny size is useful for live performances and installations where a hidden camera is desirable. Unlike the larger F200 model, the R200 does not have finger/hand tracking and doesn’t support "Marker Tracking." TouchDesigner supports both the F200 and the R200 Intel RealSense cameras.

To quote from the TouchDesigner web page, "TouchDesigner is revolutionary software platform which enables artists and designers to connect with their media in an open and freeform environment. Perfect for interactive multimedia projects that use video, audio, 3D, controller inputs, internet and database data, DMX lighting, environmental sensors, or basically anything you can imagine, TouchDesigner offers a high performance playground for blending these elements in infinitely customizable ways."

I asked Malcolm Bechard, senior developer at Derivative, to comment on using the Intel RealSense camera with TouchDesigner:

"Using TouchDesigner’s procedural node-based architecture, Intel RealSense camera data can be immediately brought in, visualized, and then connected to other nodes without spending any time coding. Ideas can be quickly prototyped and developed with an instant-feedback loop.Being a native node in TouchDesigner means there is no need to shutdown/recompile an application for each iteration of development.The Intel RealSense camera augments TouchDesigner capabilities by giving the users a large array of pre-made modules such as gesture, hand tracking, face tracking and image (depth) data, with which they can build interactions. There is no need to infer things such as gestures by analyzing the lower-level hand data; it’s already done for the user."

Using the Intel® RealSense™ Camera in TouchDesigner

TouchDesigner is a node-based platform/program that uses Python* as its main scripting language. There are five distinct categories of nodes that perform different operations and functions: TOP nodes (textures), SOP nodes (geometry), CHOP nodes (animation/audio data), DAT nodes (tables and text) and COMP nodes (3D Geometry nodes and nodes for building 2D control panels), and MAT nodes (materials). The programmers at TouchDesigner consulting with Intel programmers designed two special nodes: the Intel RealSense camera TOP node and the Intel RealSense camera CHOP node to integrate the Intel RealSense camera into the program.

Note: This article is aimed at those familiar with using TouchDesigner and its interface. If you are unfamiliar with TouchDesigner and plan to follow along with this article step-by-step, I recommend that you first review some of the documentation and videos available here:

Learning TouchDesigner

Note: When using the Intel RealSense camera, it is important to pay attention to its range for best results. On this Intel web page you will find the range of each camera and best operating practices for using it.

Intel RealSense Camera TOP Node

The TOP nodes in TouchDesigner perform many of the same operations found in a traditional compositing program. The Intel RealSense camera TOP node adds to these capabilities utilizing the 2D and 3D data feed that the Intel RealSense camera feeds into it. The Intel RealSense camera TOP node has a number of setup settings to acquire different forms of data.

• Color. The video from the Intel RealSense camera color sensor.
• Depth. A calculation of the depth of each pixel. 0 means the pixel is 0 meters from the camera, and 1 means the pixel is the maximum distance or more from the camera.
• Raw depth. Values taken directly from the Intel® RealSense™ SDK. Once again 0 means 1 meter from the camera and 1 is the maximum range or more away from the camera.
• Visualized depth. A gray-scale image from the Intel RealSense SDK that can help you visualize the depth. It cannot be used to actually determine a pixel’s exact distance from the camera.
• Depth to color UV map. The UV values from a 32-bit floating RG texture (note, no blue) that are needed to remap the depth image to line up with the color image. You can use the Remap TOP node to align the images to match.
• Color to depth UV map. The UV values from a 32-bit floating RG texture (note, no blue) that are needed to remap the color image to line up with the depth image. You can use the Remap TOP node to align the two.
• Infrared. The raw video from the infrared sensor of the Intel RealSense camera.
• Point cloud. Literally a cloud of points in 3D space (x, y, and z coordinates) or data points created by the scanner of the Intel RealSense camera.
• Point cloud color UVs. Can be used to get each point’s color from the color image stream.

Note: You can download that toe file, RealSensePointCloudForArticle.toe, to use as a simple beginning template for creating a 3D animated geometry from the data of the Intel RealSense camera. This file can be modified and changed in many ways. Together, the three Intel RealSense camera TOP nodes—the Point Cloud, the Color, and the Point Cloud Color UVs—can create a 3D geometry composed of points (particles) with the color image mapped onto it. This creates many exciting possibilities.

Point Cloud Geometry. This is an animated geometry made using the Intel RealSense camera. This technique would be exciting to use in a live performance. The audio of the character speaking could be added as well. TouchDesigner can also use the data from audio to create real-time animations.

Intel RealSense Camera CHOP Node

Note: There is also an Intel RealSense camera CHOP node that controls the 3D tracking/position data that we will discuss in Part 2 of this article.

Demo 1: Using the Intel RealSense Camera TOP Node

Click on the button on top of the article to get the First TOP Demo: settingUpRealNode2b_FINAL.toe

Demo 1, part 1: You will learn how to set up the Intel RealSense camera TOP node and then connect it to other TOP nodes.

1. Open the Add Operator/OP Create dialog window.
2. Under the TOP section, click RealSense.
3. On the Setup parameters page for the Intel RealSense camera TOP node, for Image select Color from the drop-down menu. In the Intel RealSense camera TOP node, the image of what the camera is pointing to shows up, just as in a video camera.
4. Set the resolution of the Intel RealSense Camera to 1920 by 1080.
    

   
   The Intel RealSense camera TOP node is easy to set up.

5. Create a Level TOP and connect it to the Intel RealSense camera TOP node.
6. In the Pre parameters page of the Level TOP Node, choose Invert and slide the slider to 1.
7. Connect The Level TOP node to an HSV To RGB TOP node and then connect that to a Null TOP node.

The Intel RealSense camera TOP node can be connected to other TOP nodes to create different looks and effects.

Next we will put this created image into the Phong MAT (Material) so we can texture geometries with it.

Using the Intel RealSense Camera Data to Create Textures for Geometries

Demo 1, part 2: This exercise shows you how to use the Intel RealSense camera TOP node to create textures and how to add them into a MAT node that can then be assigned to the geometry in your project.

1. Add a Geometry (geo) COMP node into your scene.
2. Add a Phong MAT node.
3. Take the Null TOP node and drag it onto the Color Map parameter of your Phong MAT node.
    

   
   The Phong MAT using the Intel RealSense camera data for its Color Map parameter.

4. On the Render parameter page of your Geo COMP for the Material parameter add type phong1 to make it use the phong1 node as its material.
    

   
   The Phong MAT using the Intel RealSense camera data for its Color Map added into the Render/Material parameter of the Geo COMP node.

Creating the Box SOP and Texturing it with the Just Created Phong Shader

Demo 1, part 3: You will learn how to assign the Phong MAT shader you created using the Intel RealSense camera data to a box Geometry SOP.

1. Go into the geo1 node to its child level, (/project1/geo1).
2. Create a Box SOP node, a Texture SOP node, and a Material SOP node.
3. Delete the Torus SOP node that was there and connect the box1 node to the texture1 node and the material1 node.
4. In the Material parameter of the material1 node enter: ../phong1 which will refer it to the phong1 MAT node you created in the parent level.
5. To put the texture on each face of the box, in the parameters of the texture1 node, Texture/Texture Type, put face and set the Texture/Offset put .5 .5 .5.
    

   
   At the child level of the geo1 COMP node, the Box SOP node, the Texture SOP node, and the Material SOP node are connected. The Material SOP is now getting its texture info from the phong1 MAT node which is at the parent level. ( …/phong1).

Animating and Instancing the Box Geometry

Demo 1, part 4: You will learn how to rotate a Geometry SOP using the Transform SOP node and a simple expression. Then you will learn how to instance the Box geometry. We will end up with a screen full of rotating boxes with the textures from the Intel RealSense camera TOP node on them.

1. To animate the box rotating on the x-axis, insert a Transform SOP node after the Texture SOP node.
2. Put an expression into the x component (first field) of the Rotate parameter in the transform1 SOP node. This expression is not dependent on the frames so it will keep going and not start repeating when the frames on the timeline run out. I multiplied by 10 to increase the speed: absTime.seconds*10
    

   
   Here you can see how the cube is rotating.

3. To make the boxes, go up to the parent level (/project1) and in the Instance page parameters of the geo1 COMP node, for Instancing change it to On.
4. Add a Grid SOP node and a SOP to the DAT node.
5. Set the grid parameters to 10 Rows and 10 Columns and the size to 20 and 20.
6. In the SOP to DAT node parameters, for SOP put grid1 and make sure Extract is to set Points.
7. In the Instance page parameters of the geo1 COMP, for Instance CHOP/DAT enter: sopto1.
8. Fill in the TX, TY, and TZ parameters with P(0), P(1), and P(2) respectively to specify which columns from the sopto1 node to use for the instance positions.
    

   
   Click on the button on top of the article to download this .toe file to see what we have done so far in this first Intel RealSense camera TOP demo.

   TOP_Demo1_forArticle.toe

9. If you prefer to see the image in the Intel RealSense camera unfiltered, disconnect or bypass the Level TOP node and the HSV to RGB TOP node.
    

Rendering or Performing the Animation Live

Demo 1, part 5: You will learn how to set up a scene to be rendered and either performed live or rendered out as a movie file.

1. To render the project, add in a Camera COMP node, a Light COMP node, and a Render TOP node. By default the camera will render all the Geometry components in the scene.
2. Translate your camera about 20 units back on the z-axis. Leave the light at the default setting.
3. Set the resolution of the render to 1920 by 1080. By default the background of a render is transparent (alpha of 0).
4. To make this an opaque black behind the squares, add in a Constant TOP node and change the Color to 0,0,0 so it is black while leaving the Alpha as 1. You can choose another color if you want.
5. Add in an Over TOP node and connect the Render TOP node to the first hook up and the Constant TOP node to the second hook up. This makes the background pixels of the render (0, 0, 0, 1), which is no longer transparent.

Another way to change the alpha of a TOP to 1 is to use a Reorder TOP and set its Output Alpha parameter to Input 1 and One.

Shows the rendered scene with the background being set to opaque black.

Here you can see the screen full of the textured rotating cubes.

If you prefer to render out the animation instead of playing it in real time in a performance you must choose the Export movie Dialog box under file in the top bar of the TouchDesigner program. In the parameter for the TOP Video, enter null2 for this particular example. Otherwise enter any TOP node that you want to render.

Here is the Export Movie panel, and null2 has been pulled into it. If I had an audio CHOP to go along with it, I would pull or place that into the CHOP Audio slot directly under where I put null2.

Demo 1, part 6: One of the things that makes TouchDesigner a special platform is the ability to do real-time performance animations with it. This makes it especially good when paired with the Intel RealSense Camera.

1. Add a Window COMP node and in the operator parameter enter your null2 TOP node.
2. Set the resolution to 1920 by 1080.
3. Choose the Monitor you want in the Location parameter. The Window COMP node lets you perform the entire animation in real time projected onto the monitor you choose. Using the Window COMP node you can specify the monitor or projector you want the performance to be played from.
    

   
   You can create as many Window COMP nodes as you need to direct the output to other monitors.

Demo 2: Using the Intel RealSense Camera TOP Node Depth Data

The Intel RealSense camera TOP node has a number of other settings that are useful for creating textures and animation.

In demo 2, we use the depth data to apply a blur on an image based on depth data from the camera. Click on the button on top of the article to get this file: RealSenseDepthBlur.toe

First, create an Intel RealSense camera TOP and set its Image parameter to Depth. The depth image has pixels that are 0 (black) if they are close to the camera and 1 (white) if they are far away from the camera. The range of the pixel values is controlled by the Max Depth parameter which is specified in Meters. By default it has a value of 5 which means pixels 5 or more meters from the camera will be white. A pixel with a value of 0.5 will be 2.5 meters from the camera. Depending on how far the camera is from you changing this value to something smaller may be good. For this example we’ve changed it to 1.5 meters.

Next we want to process the depth a bit to remove objects outside our range interest, which we will do using a Threshold TOP.

1. Create a Threshold TOP and connect it to the realsense1 node. We want to cull out pixels that beyond a certain distance from the camera so set the Comparator parameter to Greater and set the Threshold parameter to 0.8. This makes pixels that are greater than 0.8 (which is 1.2 meters or greater if we have Max Depth in the Intel RealSense camera TOP set to 1.5), become 0 and all other pixels become 1.
    

2. Create a Multiply TOP and connect the realsense1 node to the first input and the thresh1 node to the 2nd input. Multiplying the pixels we want by 1 will leave them as-is and others by 0 make them back. The multiply1 node now has only pixels greater than 0 for the part of the image you want to control the blur we will do next.
3. Create a Movie File in TOP, and select a new image for its File parameter. In this example we select Metter2.jpg from the TouchDesigner Samples/Map directory.
4. Create a Luma Blur TOP and connect moviefilein1 to the 1st input of lumablur1 and multiply1 to the 2nd input of lumablur1.
5. In the parameters for lumablur1 set White Value to 0.4, Black Filter Width to 20, and White Filter Width to 1. This makes pixels where the first input is 0 have a blur filter width of 20 and a pixels with a value of 0.4 or greater have a blur width of 1.
    

   
   The whole layout.

The result is an image where the pixels where the user is located are not blurred while other pixels are blurry.

The background, by putting on the display of the Luma Blur TOP shows how the image is blurred.

Demo 3: Using the Intel RealSense Camera TOP Node Depth Data with the Remap TOP Node

Click on the button on the article top to get this file: RealSenseRemap.toe

Note: The depth and color cameras of the Intel RealSense camera TOP node are in different spots in the world so their resulting images by default do not line up. For example if your hand is positioned in the middle of the color image, it won’t be in the middle of the depth image, it will either be off to the left or right a bit. The UV remap fixes this by shifting the pixels around so they align on top of each other. Notice the difference between the aligned and unaligned TOPs.

The Remap TOP aligns the depth data from the Intel RealSense camera TOP with the color data from the Intel RealSense camera TOP, using the depth to color UV data, putting them in the same world space.

Demo 4: Using Point Cloud in the Intel RealSense Camera TOP Node

Click on the button on top of the article to get this file: PointCloudLimitEx.toe

In this exercise you learn how to create animated geometry using the Intel RealSense camera TOP node point Cloud setting and the Limit SOP node. Note that this technique is different than the Point Cloud example file shown at the beginning of this article. The previous example uses GLSL shaders, which results in the ability to generate far more points, but it is more complex to do and out of the scope of this article.

1. Create a RealSense TOP node and set the parameter Image to Point Cloud.
2. Create a TOP to CHOP node and connect it to a Select CHOP node.
3. Connect the Select CHOP node to a Math CHOP node.
4. In the topto1 CHOP node parameter, TOP, enter: realsense1.
5. In the Select CHOP node parameters, Channel Names, enter r g b leaving a space between the letters.
6. In the math1 CHOP node for the Multiply parameter, enter: 4.2.
7. On the Range parameters page, for To Range, enter: 1 and 7.
8. Create a Limit SOP node.

To quote from the information on the www.derivative.ca online wiki page, "The Limit SOP creates geometry from samples fed to it by CHOPs. It creates geometry at every point in the sample. Different types of geometry can be created using the Output Type parameter on the Channels Page."

1. In the limit1 CHOP Channels parameters page, enter r in the X Channel, g in the Y Channel, and b in the Z Channel.
    

   Note: Switching the r g and b to different X Y or Z channels changes the geometry being generated. So you might want to try this later: In the Output parameter page, for Output Type select Sphere at Each Point from the drop-down. Create a SOP to DAT node. In the parameters page, for SOP put in limit1 or drag your limit1 CHOP into the parameter. Keep the default setting of Points in the Extract parameter. Create a Render TOP node, a Camera COMP node, and a Light COMP node. Create a Reorder TOP and make Output Alpha be Input 1 and One and connect it to the Render TOP.

   
   As the image in the Intel RealSense camera changes, so does the geometry. This is the final layout.

   
   Final images in the Over TOP CHOP node. By changing the order of the channels in the Limit TOP parameters you change the geometry which is based on the point cloud.

In Part 2 of this article we will discuss the Intel RealSense camera CHOP and how to create content both rendered and in real-time for performances, Full Dome shows, and VR. We will also show how to use the Oculus Rift CHOP node.

About the Author

Audri Phillips is a visualist/3d animator based out of Los Angeles, with a wide range of experience that includes over 25 years working in the visual effects/entertainment industry in studios such as Sony, Rhythm and Hues, Digital Domain, Disney, and Dreamworks feature animation. Starting out as a painter she was quickly drawn to time based art. Always interested in using new tools she has been a pioneer of using computer animation/art in experimental film work including immersive performances. Now she has taken her talents into the creation of VR. Samsung recently curated her work into their new Gear Indie Milk VR channel.

Her latest immersive work/animations include: Multi Media Animations for "Implosion a Dance Festival" 2015 at the Los Angeles Theater Center, 3 Full dome Concerts in the Vortex Immersion dome, one with the well-known composer/musician Steve Roach. She has a fourth upcoming fulldome concert, "Relentless Universe", on November 7th, 2015. She also created animated content for the dome show for the TV series, “Constantine” shown at the 2014 Comic-Con convention. Several of her Fulldome pieces, “Migrations” and “Relentless Beauty”, have been juried into "Currents", The Santa Fe International New Media Festival, and Jena FullDome Festival in Germany. She exhibits in the Young Projects gallery in Los Angeles.

She writes online content and a blog for Intel. Audri is an Adjunct professor at Woodbury University, a founding member and leader of the Los Angeles Abstract Film Group, founder of the Hybrid Reality Studio (dedicated to creating VR content), a board member of the Iota Center, and she is also an exhibiting member of the LA Art Lab. In 2011 Audri became a resident artist of Vortex Immersion Media and the c3: CreateLAB.

Deep neural network (DNN) training is computationally intensive and can take days or weeks on modern computing platforms. In the recent article, Single-node Caffe Scoring and Training on Intel® Xeon® E5 Family, we demonstrated a tenfold performance increase of the Caffe* framework on the AlexNet* topology and reduced the training time to 5 days on a single node. Intel continues to deliver on the machine learning vision outlined in Pradeep Dubey’s Blog, and in this technical preview, we demonstrate how the training time for Caffe can be reduced from days to hours in a multi-node, distributed-memory environment.

Caffe is a deep learning framework developed by the Berkeley Vision and Learning Center (BVLC) and one of the most popular community frameworks for image recognition. Caffe is often used as a benchmark together with AlexNet*, a neural network topology for image recognition, and ImageNet*, a database of labeled images.

The Caffe framework does not support multi-node, distributed-memory systems by default and requires extensive changes to run on distributed-memory systems. We perform strong scaling of the synchronous minibatch stochastic gradient descent (SGD) algorithm with the help of Intel® MPI Library. Computation for one iteration is scaled across multiple nodes, such that the multi-threaded multi-node parallel implementation is equivalent to the single-node, single-threaded serial implementation.

We use three approaches—data parallelism, model parallelism, and hybrid parallelism—to scale computation. Model parallelism refers to partitioning the model or weights into nodes, such that parts of weights are owned by a given node and each node processes all the data points in a minibatch. This requires communication of the activations and gradients of activations, unlike communication of weights and weight gradients, as is the case with data parallelism.

With this additional level of distributed parallelization, we trained AlexNet on the full ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC-2012) dataset and reached 80% top-5 accuracy in just over 5 hours on a 64-node cluster of systems based on Intel® Xeon® processor E5 family.

Getting Started

While we are working to incorporate the new functionality outlined in this article into future versions of Intel^® Math Kernel Library (Intel^® MKL) and Intel^® Data Analytics Acceleration Library (Intel^® DAAL), you can use the technology preview package attached to this article to reproduce the demonstrated performance results and even train AlexNet on your own dataset. The preview includes both the single-node and the multi-node implementations. Note that the current implementation is limited to the AlexNet topology and may not work with other popular DNN topologies.

The package supports the AlexNet topology and introduces the ‘intel_alexnet’ and ‘mpi_intel_alexnet’ models, which are similar to ‘bvlc_alexnet’ with the addition of two new ‘IntelPack’ and ‘IntelUnpack’ layers, as well as the optimized convolution, pooling, normalization layers, and MPI-based implementations for all these layers. We also changed the validation parameters to facilitate vectorization by increasing the validation minibatch size from 50 to 256 and reducing the number of test iterations from 1,000 to 200, thus keeping constant the number of images used in the validation run. The package contains the ‘intel_alexnet’ model in these folders:

• models/intel_alexnet/deploy.prototxt
• models/intel_alexnet/solver.prototxt
• models/intel_alexnet/train_val.prototxt.

• models/mpi_intel_alexnet/deploy.prototxt
• models/mpi_intel_alexnet/solver.prototxt
• models/mpi_intel_alexnet/train_val.prototxt.
• models/mpi_intel_alexnet/train_val_shared_db.prototxt
• models/mpi_intel_alexnet/train_val_split_db.prototxt

Both the ’intel_alexnet’ and the ’mpi_intel_alexnet’ models allow you to train and test the ILSVRC-2012 training set.

To start working with the package, ensure that all the regular Caffe dependencies and Intel software tools listed in the System Requirements and Limitations section are installed on your system.

Running on Single Node

1. Unpack the package.
2. Specify the paths to the database, snapshot location, and image mean file in these ‘intel_alexnet’ model files:
   • models/intel_alexnet/deploy.prototxt
   • models/intel_alexnet/solver.prototxt
   • models/intel_alexnet/train_val.prototxt
3. Set up a runtime environment for the software tools listed in the System Requirements and Limitations section.
4. Add the path to ./build/lib/libcaffe.so to the LD_LIBRARY_PATH environment variable.
5. Set the threading environment as follows:
   $> export OMP_NUM_THREADS=<N_processors * N_cores>
   $> export KMP_AFFINITY=compact,granularity=fine

Note: OMP_NUM_THREADS must be an even number equal to at least 2.

1. Run timing on a single node using this command:
   $> ./build/tools/caffe time \
          -iterations <number of iterations> \
          --model=models/intel_alexnet/train_val.prototxt
2. Run training on a single node using this command:
   $> ./build/tools/caffe train \
          --solver=models/intel_alexnet/solver.prototxt

Running on Cluster

1. Unpack the package.
2. Set up a runtime environment for the software tools listed in the System Requirements and Limitations section.
3. Add the path to ./build-mpi/lib/libcaffe.so to the LD_LIBRARY_PATH environment variable.
4. Set the NP environment variable to the number of nodes to be used, as follows:

$> export NP=<number-of-mpi-ranks>

Note: the best performance is achieved with one MPI rank per node.

1. Create a node file in the root directory of the application with the name of x${NP}.hosts. For instance, for IBM* Platform LSF*, run the following command:

$> cat $PBS_NODEFILE > x${NP}.hosts

1. Specify the paths to the database, snapshot location, and image mean file in the following ‘mpi_intel_alexnet’ model files:
   • models/mpi_intel_alexnet/deploy.prototxt,
   • models/mpi_intel_alexnet/solver.prototxt,
   • models/mpi_intel_alexnet/train_val_shared_db.prototxt

Note: on some system configurations, performance of a shared-disk system may become a bottleneck. In this case, pre-distributing the image database to compute nodes is recommended to achieve best performance results. Refer to the readme files included with the package for instructions.

1. Set the threading environment as follows:

$> export OMP_NUM_THREADS=<N_processors * N_cores>
$> export KMP_AFFINITY=compact,granularity=fine

Note: OMP_NUM_THREADS must be an even number equal to at least 2.

1. Run timing using this command:
   $> mpirun -nodefile x${NP}.hosts -n $NP -ppn 1 -prepend-rank \

         ./build/tools/caffe time \

         -iterations <number of iterations> \

        --model=models/mpi_intel_alexnet/train_val.prototxt

1. Run training using this command:
   $> mpirun -nodefile x${NP}.hosts -n $NP -ppn 1 -prepend-rank \

         ./build-mpi/tools/caffe train \

         --solver=models/mpi_intel_alexnet/solver.prototxt

System Requirements and Limitations

The package has the same software dependencies as non-optimized Caffe:

• boost* 1.53.0
• opencv* 2.4.9
• protobuf* 3.0.0-beta1
• glog* 0.3.4
• gflags* 2.1.2
• lmdb* 0.9.16
• leveldb* 1.18
• hdf5* 1.8.15
• Red Hat Enterprise Linux* 6.5 or higher

Intel software tools:

• Intel MKL 11.3 or higher
• Intel MPI Library 5.0

Hardware compatibility:

• 4^th Generation Intel® Core™ Processor (code name Haswell)

This software was validated with the AlexNet topology only and may not work with other configurations.

Support

Please direct questions and comments on this package to mailto:intel.mkl@intel.com.

In a world where internet video use is skyrocketing and consumers expect High Definition and ultra-high definition (UHD) 4K viewing anytime, anywhere and on any device, excel with Intel in delivering live and on-demand video faster, more efficiently, and at higher quality through the latest media acceleration technologies. Get the most performance from your media platform, and accelerate to 4K/UHD and HEVC, while reducing infrastructure and development costs.

Attend this free online webinar, Video Transcode Solutions on Dec. 1, 9 a.m. (Pacific), to learn about new media acceleration and Intel graphics technologies. Offer your cloud and communication service provider customers a customizable solution that can:

• Deliver fast video transcoding into multiple formats and bit rates in less time
• Reduce the amount of storage needed for multiple formats through higher compression processing and offering multiple rate control techniques
• Allow for real-time transcoding into multiple formats from the stored format, reducing the need to store all possible media formats
• Reduce the amount of network bandwidth needed (lower bit rates) at better video quality by compressing the video appropriately prior to transmission

Video Transcode Solutions: Simple, Fast Efficient
Online Webinar | Dec. 1, 2015 - 9 a.m. (Pacific)    
     

See how Intel can help you innovate and bring new media solutions quicker to market. Webinar is for cloud media solutions and video streaming/conferencing providers, media/graphics developers, broadcast/datacenter engineers, and IT/business decision-makers.

Speakers

Shantanu Gupta, Intel Media Accelerator Products Director
Shantanu has held leadership roles in technology/solutions marketing, integration, product design/development and more areas - for 27 years at Intel.

Mark J. Buxton, Intel Media Development Products Director
Mark has more than 20 years experience leading video standards development and Intel’s media development product efforts, including products such at Intel® Media Server Studio, Intel® Video Pro Analyzer, and Intel® Stress Bitstreams and Encoder

Introduction

The Intel® RealSense™ Depth Camera Manager (DCM) is intended to expose interfaces to streaming video from the Intel® RealSense™ camera F200 and R200, for both color and depth data streams. The camera service allows multiple Intel® RealSense™ SDK applications and a single non-SDK application to access data from the camera simultaneously, without blocking each other. Without the camera service, only one application can access the data from the camera at a time to assure that the correct data is received.

DCM Functionality

The DCM is the primary interface between the Intel RealSense camera and SDK clients via the Intel RealSense SDK APIs. The DCM exposes and manipulates all extended 2D and 3D camera capabilities to the client system. It provides a compatible interface to standard video application within the DCM environment via a virtual imaging device driver. It also manages camera control, access policy, and power management when multiple applications access the DCM. For these DCM functionalities to work properly, the DCM must be downloaded from Intel and installed on the platform that is equipped with an Intel RealSense camera. Visit https://downloadcenter.intel.com/download/25044 to download F200 and http://registrationcenter-download.intel.com/akdlm/irc_nas/7787/intel_rs_dcm_r200_2.0.3.39488.exe to download R200 DCM for Windows* 8.1 and Windows 10. The functionality of the DCM applies to different Intel RealSense camera models such as F200 and R200.

F200 Camera Model

The Intel RealSense camera F200 is the first generation of front-facing 3D cameras based on coded light depth technology. The camera implements an infrared (IR) laser projector system, VGA infrared (IR) camera, and a 2MP color camera with integrated ISP. This camera enables new platform usages by providing synchronized color, depth, and IR video stream data to the client system. The effective range of the depth solution from the camera is optimized from 0.2 to 1.0m for use indoors.

R200 Camera Model

The Intel RealSense camera R200 is the first generation of rear-facing 3D cameras based on active stereoscopic depth technology. The camera implements an IR laser projector, VGA stereoscopic IR cameras, and a 2MP color camera with integrated ISP. With synchronized color and infrared depth sensing features, the camera enables a wide variety of exciting new platform usages. The depth usage range of the camera depends upon the module and the lighting. The indoor range is up to 3 meters and the outdoor range is up to 10 meters.

Figure 1: DCM Model – High level view

Hardware Requirements

For details on system requirements and supported operating systems for F200 and R200, see https://software.intel.com/en-us/RealSense/Devkit/

DCM Components

There are two DCM components: DCM service and DCM virtual driver.

The DCM service runs on the client machine and controls multiple application requests to operate the managed cameras. The DCM service also dispatches multiple access requests from several applications accessing the same video stream. The DCM service runs at startup and allows multiple clients applications to connect to it. The DCM Service interfaces to the camera through the Camera DLL and is the primary camera interface for all application types. The Camera DLL is camera-model specific and extends hardware functionality for each camera. Below is an example of the task manager of the system that has DCMs for F200 and R200 installed.

Figure 2: The DCM Service runs at startup

The DCM virtual driver is a virtual AVStream device driver that supports a compatible interface into the DCM for standard video applications. This virtual driver allows standard video applications to access the camera concurrently with multiple SDK applications.
 

Detecting the DCM Version

Go to the shortcut “Intel® RealSense™ SDK Gold” from the desktop, then “Samples\Sample binaries” or C:\Program Files (x86)\Intel\RSSDK\bin\win32 directory and open sdk_info. The Camera tab shows the DCM service version and other information about the cameras that are installed in the platform. For testing and development purposes, multiple major versions for the DCM can be installed on a platform. During runtime, only one camera—whether the same model or a different one—can be connected to the platform at a time.

Figure 3: RealSense SDK information

Troubleshooting

If the Intel RealSense camera does not stream data correctly:

• Make sure that the DCM service exists and is running, as shown in Figure 2.

• Check control panel to make sure that the app installed the Intel RealSense SDK Runtime during installation.

• Make sure that the camera is connected properly.

Switching Cameras between DCM Runtimes

An SDK client can support different camera models through their respective DCM runtime. The SDK client must close any access to one camera model before switching to the next camera model. Multiple concurrent accesses from the SDK client to multiple camera models are not allowed. If an SDK client enables simultaneous access between multiple camera models, unknown behaviors are likely to occur.

Uninstallation

Before installing the new version of the DCM, uninstall any existing versions. Once you launch the DCM driver on the system that has an existing DCM installed, an uninstaller menu will prompt the uninstall option and other options.

• Modify. Edit currently installed feature or feature settings.

• Repair. Fix missing or corrupt files, shortcuts, or registry entries.

• Remove. Remove the DCM from the system registries and directory structure.

   

Summary

The Intel RealSense Depth Camera Manager is the primary interface between the Intel RealSense camera and the Intel RealSense SDK clients. It communicates with the Intel RealSense camera through the camera DLL.

Helpful References

Here is a collection of useful references for the Intel RealSense DCM and SDK, including release notes and how to download and update the software.

• Developer Zone

• How to Update Intel® RealSense™ Depth Camera Manager (DCM)

• Intel® RealSense™ Depth Camera Manager (DCM) Overview

• Intel® RealSense™ Depth Camera Manager (DCM) for F200

• Intel® RealSense™ Depth Camera Manager (DCM) for F200

• What’s New in the Intel® RealSense^TM SDK R4 (v.6.0)

About the Author

Nancy Le is a software engineer at Intel Corporation in the Software and Services Group working on Intel® Atom™ processor scale-enabling projects.

Notices

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.

The products and services described may contain defects or errors known as errata which may cause deviations from published specifications. Current characterized errata are available on request.

Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting www.intel.com/design/literature.htm.

Intel, the Intel logo, and Intel RealSense are trademarks of Intel Corporation in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others.

© 2015 Intel Corporation.

Jacob Pennock from Livid Interactive imparts some valuable advice on working with Intel® RealSense™ technology and Unity* 5 in this series of seven videos. See below for a description of each video.

Note: In some of the videos, Jacob references a video that describes the SDK’s emotion module. That video is no longer available because the emotion module has been deprecated, and starting in the Intel® RealSense™ SDK R5, it has been removed.

How to Set Up the Intel® RealSense™ SDK Toolkit in Unity* 5

Jacob describes how to import the Intel RealSense SDK Toolkit into Unity 5 and test your installation with a gesture-based drag-and-stretch code sample from the Intel RealSense SDK. Note: if you’ve installed version R4 or above of the Intel RealSense SDK, you can disregard the instructions to swap out 32-bit toolkit binaries and replace with 64-bit binaries, because starting with R4, the toolkit ships with 64-bit binaries built for Unity 5.

Optimizations for the Intel® RealSense™ SDK Toolkit - Part 1

In this two-part video on optimizations, Jacob highlights aspects of the Intel RealSense Toolkit where performance is not adequate for a production application. In Part 1 he describes how to significantly improve frame rate performance with a change to SenseToolkitMgr.Update(). He also shows how to fix a mirroring problem with one of the toolkit samples.

Optimizations for the Intel® RealSense™ SDK Toolkit- Part 2

In Part 2, Jacob demonstrates how to fix a performance problem with toolkit method DrawImages.Update(). In the video he creates a new, more memory-efficient method to replace an Intel private method that has a large memory footprint.

How to Use the Intel® RealSense™ SDK in Unity* 5 Directly (without the toolkit)

In this video Jacob details how to access the Intel RealSense SDK directly from inside your Unity 5 C# code. The Intel RealSense Toolkit can be useful for quick testing, but developers will get better results for most applications by building customized interactions with the Intel RealSense SDK. He goes over the basic steps required to create customized interactions.

Intel® RealSense™ SDK + Unity* 5 Pro Tip: Monitor the Frame Rate

Jacob has learned over the past 2 years of working with Intel RealSense technology that it is helpful to measure the frame rate at which RealSense is returning data. In this video he goes over one way to accomplish this.

Alternative Hand Tracking Modes with the Intel® RealSense™ SDK and Unity* 5

The Intel RealSense SDK includes a couple of skeletal hand tracking examples. In this video Jacob demonstrates another hand tracking technique using the Extremities mode and the segmentation image it produces. This is much faster than using skeletal hand tracking.

Intel® RealSense™ SDK + Unity* 5 Pro Tip: Displaying Segmentation Images

Here Jacob takes the low-resolution hand segmentation images he created in the previous video and scales them up in size to something that is presentable to the user without looking pixelated.

Intel® RealSense™ SDK + Unity*: 5 Pro Tip: Update in a Separate Thread

Jacob describes his new Asus laptop with an embedded Intel® RealSense™ camera, and demonstrates how to improve performance of your application by using a separate thread for most Intel RealSense SDK interactions.

About the Author

Jacob Pennock is an Intel® RealSense™ expert from the Bay Area. He is Senior Creative Developer, Helios Interactive Technologies & Chief Game Designer / Lead Developer, Livid Interactive.

Download PDF [PDF 1 MB]

Contents

Why Biometric Recognition is Better
How Biometric Recognition Works
The Attack Model
How Intel® Hardware-based Security Technologies Improve the Security of Biometrics Recognition

1. Trusted Execution Environment with Intel® Software Guard Extensions
   • What is TEE?
   • Intel® Software Guard Extensions Technology Overview
   • How to Harden Biometric Recognition Function by Intel® Software Guard Extensions Technology?
2. Memory Protection Scheme with Virtual Machine Extensions
   • The Weakness of the Legacy Software-Based Memory Protection
   • Memory Protection Based on Virtual Machine Extensions
3. Multiple Factor Authentication with Intel® Identity Protection Technology with One-Time Password

Link to Part 2
References

Why Biometric Recognition is Better

The security model of “Username/Password” has been used as a user’s identity certificate for years. When people need to prove that they are authorized users of a service (the usual process is to log in to a computer or an online service, such as social media, or online banking), they input their username and password. The disadvantages of this security model are obvious for a number of reasons, including:

1. Simple passwords such as “123456” or “hello” can be cracked by a brute-force attack or a dictionary attack.
2. A complicated password is hard to remember.
3. Many people might use the same password across multiple sites.
4. If a person forgets the password, after providing some other identification, he or she can reset it.

Figure 1. Password login scheme.

To improve the strength of passwords and the user experience, more and more service providers are beginning to use biometric identification technology as the password. With this technology, people don’t need to remember their passwords. Instead their voice, face, fingerprint, or iris is used as the identifying factor. Biometric identification factors are somewhat different from the traditional username/password security model factors:

• Biometrics can be used to derive a long and complicate password, which offers greater security to withstand a brute-force attack.
• Biometrics require more security protection to a biometric recognition application developer, because biological information is part of the human body and cannot be changed easily. If biometric information is stolen, it is hard for a user to revoke his or her biometric password. An attacker can duplicate a fake human part by using the stolen biometrics and use it to pass the biometric check on other user registered accounts in future.
• Some biological characteristics such as face and voice have a high false acceptance rate. So a biometrics recognition system usually uses multi-biometric factor authentication to improve the recognition accuracy.
• Some biometrics characteristics can be duplicated, like a recorded voice, a printed photo of a face, or a gelatin finger from a fingerprint. It is important to add a vitality detection module into the biometric recognition system to identify whether the biometrics information is from a live person or from a replicate object.

How Biometric Recognition Works

The basic flow of a biometric recognition application has five steps:

1. Biometric information is collected by the sensor, which is connected through the I/O port.
2. The output data format and speed is controlled by the specific device driver. Data is processed via the driver to meet the OS requirement at Ring-0 level and then sent to the biometric verification app, which is running at Ring-3.
3. Once the app gets the data, it does some preprocessing work and extracts the feature points from the data.
4. Next, the extracted feature points are sent to the pattern matcher and compared with registered biometric patterns in the database.
5. Once the pattern matches with one of the registered patterns, the matcher sends the MATCH message, and the UI procedure will display that the user is logged in correctly and show the corresponding premier content to the user.

Figure 2. The flow chart of a biometric recognition program.

The Attack Model

In a biometric-based authentication system, the most valuable data to an attacker is the user’s biometric pattern. This pattern could be the raw data from a sensor, the extracted feature point set in memory, or the registered biological pattern stored in the database.

In general, if the biometric recognition application is designed without proper security protection, the attacker could retrieve the raw data or feature point set from memory via runtime attack by using a rootkit or malware. The attacker could also launch an offline attack to get the registered biological pattern if the registration template is stored at the local storage of the device. Moreover, the attacker could sniff the data stream from a data bus between the processor and the sensor or by using a camera or microphone near the user to get some biometric data like face pictures or voice samples for a further replay attack.

Figure 3. Possible attacks on a biometrics recognition application.

From the perspective of a biometrics recognition service developer, the design philosophy of the application should provide end-to-end protection to keep a user’s privacy safe. This includes:

• Provide a trusted running environment to keep the integrity of the application code segment.
• Protect the memory region, which contains the biometric pattern, from access by other applications.
• Keep the sensitive data with strong encryption when it is in memory/local storage (or exchange the secret data between other applications or the network server).

How Intel® Hardware-based Security Technologies Improve the Security of Biometrics Recognition

Intel’s platform offers various hardware-based security technologies to satisfy the security requirements for biometric verification applications.

1. Trusted Execution Environment with Intel® Software Guard Extensions

Biometric recognition technology is being used more and more widely because of its security. Because the technology is based on the unique characteristics—face, voice, fingerprint, iris—of each dedicated person, a person’s identity is hard to steal. Biometric recognition technology takes the place of traditional password authentication and offers a good user experience.

However, with the wide use of biometric recognition technology in various consumer devices, the diversity and openness of the platform has raised some potential security threats. One threat that developers need to carefully consider is how to secure the operation of a biometric identification function on a variety of terminal devices. In particular they need to consider:

• How to securely run the biometric sampling/modeling/matching algorithm on the terminal device
• How to securely store biometric data template on the terminal device
• How to establish a secure channel link between the terminal device and the cloud database of biological characteristics, to complete the cloud authentication and other operations

Developers can rely on Trusted Execution Environment (TEE) technology to build an effective hardened solution.

What is TEE?

TEE is an isolated, trusted execution environment isolated from the Rich Execution Environment (REE).

According to the Global Platform TEE System Architecture specification¹, at the highest level, a TEE is an environment where the following are true:

• Any code executing inside the TEE is trusted in authenticity and integrity.
• The other assets are also protected in confidentiality.
  • The TEE shall resist all known remote and software attacks, and a set of external hardware attacks.
• Both assets and code are protected from unauthorized tracing and control through debug and test features.

Intel® Software Guard Extensions Technology Overview

Intel® Software Guard Extensions (Intel® SGX) enables SW developers to develop and deploy secure applications on PC open platforms. It is a set of new instructions and memory access changes added to Intel® architecture.

Intel® SGX operates by allocating hardware-protected memory where code and data reside. The protected memory area is called an enclave. Data within the enclave memory can only be accessed by the code that also resides within the enclave memory space. Enclave code can be invoked via special instructions. An enclave can be built and loaded as a Windows* DLL.

Figure 4. Protected execution environment embedded in a process.

One Intel® SGX technology-enabled application is built as an untrusted part and a trusted part following the Intel® SGX design framework². When the application is running, it calls Intel® SGX special instructions to create an enclave, which is placed in trusted memory. When the trusted function is called, the code is running inside the enclave, and the relevant data can be seen in clear text only inside the enclave. Any external access to this data is denied. After the trusted function returns, the enclave data remains in trusted memory.

Figure 5. Intel® Software Guard Extensions technology-enabled application execution flow.

The objective of this Intel® technology is to enable a high-level protection of secrets. The application gains the ability to defend its own secrets thanks to Intel® SGX technology. Sensitive data is protected within applications. The attack surface, or TCB (Trusted Computing Base), is minimized as application self and processor. Even the malware that subverts OS/VMM, BIOS, drivers, etc. cannot steal the application secrets.

Figure 6. Reduced attack surface with Intel® Software Guard Extensions.

How to HardenBiometric Recognition Function by Intel® Software Guard Extensions Technology?

Before we discuss the security solution proposal for biometric recognition, we should address which factors should be protected during the process:

• The user’s private biometric characteristics data should be handled carefully in application, at rest, and in flight.
• The biometric operation algorithm, including sampling, modeling, and matching, should be protected against viruses and malware. The output result data should not be tampered with.

We proposed the architecture shown in Figure 7.

Figure 7. Hardened biometric recognition function by Intel® Software Guard Extensions.

The biometric sampling/modeling/matching algorithm is hosted inside the Intel® SGX enclave, the trusted part of the client, and is responsible for operating on the biometric characteristics data. Its runtime confidentiality and integrity is guaranteed. This type of algorithm is normally software implementation.. Normal software implementation may be tampered with at runtime by viruses and malware. But in this architecture, at runtime, the protected portion is loaded into an enclave where its code and data is measured. Once the application’s code and data is loaded into an enclave, it is protected against all external software access. So the biometric operation algorithm can be trusted. Beyond the security properties, the enclave environment offers scalability and performance associated with execution on the main CPU of an open platform. So it is helpful for the performance sensitive scenario, especially the biometric recognition function.

Intel® SGX technology provides a function to encrypt and integrity-protect enclave secrets to store them outside the enclave, such as on disk, and the application can reuse this data later. Data can be sealed against an enclave using a hardware-derived Seal Key. The Seal Key is unique to the CPU and the specific enclave environment. Combined with other services, like Monotonic Counter or Trusted Time, which are provided by the Intel® SGX Platform Software, the solution can be used to protect against various attack techniques. The Monotonic Counter can be used to implement replay-protected policy, and the Trusted Time can be used to enforce a time-based policy. Both of them are in a form of Sealed Data. The enclave is responsible for performing the encryption with an algorithm of its choice; in other words, the developer can choose any encryption framework according to their system security requirement. So we can keep the user’s privacy biometric characteristics data to be handled only within the enclave and make sure that its raw data cannot be exposed to the untrusted part out of the enclave.

Sometimes the client biometric recognition function needs to connect to the remote back-end server to do authentication in the cloud database instead of locally. Using Intel® SGX attestation capabilities, the client authentication module authenticates the client platform and user’s biometric characteristics data with the remote server. Attestation is the process of demonstrating that a piece of software has been properly instantiated on the platform. In Intel® SGX it is the mechanism by which another party can gain confidence that the correct software is securely running within an enclave on an enabled platform.

First, this module generates a verifiable report of the client’s identity that is bound to the platform by the CPU³. The report also includes information about the user running the biometric recognition session. The server verifies the report to ensure that it is communicating with a device that is enabled with Intel® SGX. The client and server engage in a one-time provisioning protocol that results in application secrets being securely sealed to the client platform, using Intel® SGX sealing capabilities.

These secrets, which can only be unsealed by the application that sealed it, are used to establish secure sessions with the server in the future, without the need for constantly proving the identity of the client platform. Such secrets can be salt, encryption key, policy, certificate, etc. After that, the biometric characteristics data and authentication result can be sent through the secure communication channel between the client and server.

2. Memory Protection Scheme with Virtual Machine Extensions

Dynamic data attack is one of the most commonly known attack methodologies. Rootkit and malware can use this technique to hook a specified function and dump/modify data in the memory at runtime. In the case of biometrics recognition, malicious code can get the biometric data captured from the sensor and registered user biometric template from memory.

The Weakness of the Legacy Software-Based Memory Protection

Traditional software-based memory protection mechanisms are not reliable enough. Both the protection code and malicious code are running at the same privilege (ring-0 or ring-3). So malware can compromise the protection code easily to disable the protection.

Figure 8. Attacks can compromise the protection module and access the sensitive data buffer.

Memory Protection Based on Virtual Machine Extensions

Virtual Machine Extensions (VMX) is a set of instructions that support virtualization of processor hardware⁴. Its basic working logic is:

• Ignore the basic CPU operation like load/store, branch and ALU operations
• Monitor (trap) the privileged instructions such as MMU manipulation, I/O instructions or update the TLB
• If a privilege instruction is executed, break the execution and set CPU into VMX root mode for further processing

The following diagram shows the relationship between the hardware/OS/application with VMM mode enable/disable.

Figure 9. Different response to the system call when Virtual Machine Extensions mode is on/off.

By utilizing the hardware-based trap function of VMX, a hardware virtualization-based memory protection mechanism can protect memory in a safer and faster way⁵ The basic idea is to insert a VMM-based memory monitor module between the OS and hardware. When loading the application, build a memory map table for the trusted code region and data region. After building the table, whenever there is memory access, VMM can trap it, then compare the memory access instruction address (EIP) and memory address with the pre-built table. Then the memory protection module can identify whether this is a legal or illegal access and apply for the corresponding process.

3. Multiple Factor Authentication with Intel® Identity Protection Technology with One-Time Password

Identity theft is a growing global concern for individuals and businesses. Secure, but simple-to-use solutions are required as hackers devise new methods for obtaining usernames and passwords. Hackers never stop devising new ways to steal usernames and passwords. If you are a consumer or an everyday computer user, Intel® Identity Protection Technology (Intel® IPT) provides strong techniques for avoiding the threat of identity theft by giving you the opportunity to link your physical device to each Intel® IPT-enabled online account that you use.

Traditionally, two-factor authentication uses a one-time password (OTP) which combines something the user knows (a username and password) and something the user has (typically, a token or key fob that produces a six-digit number, valid only for a short period of time and available on demand).

In the case of Intel® IPT with OTP⁶, a unique, one-time use, six-digit number is generated every 30 seconds from an embedded processor that is tamper-proof and operates in isolation from the OS. Because the credential is protected inside the chipset, it cannot be compromised by malware or removed from the device.

Figure 10. Intel® Identity Protection Technology with one-time password authentication working flow between client and server.

If your business is already using two-factor authentication, you are already familiar with the various issues around token usability and logistics. Intel® IPT with OTP is a built-in hardware token (with your security vendor of choice) that negates the need for a separate physical token, thus simplifying the two-factor VPN log-in process for a seamless experience with virtually no delays.

With Intel® IPT with OTP on Intel® processor-based devices, Intel provides a hardware root of trust, proof of a unique Intel processor-based device, to websites, financial institutions, and network services that it is NOT malware logging into an account. Intel® IPT with OTP enabled systems offer additional identity protection and transaction verification methods that can be utilized by multifactor authentication solutions.

About the Author

Jianjun Gu is a senior application engineer in the Intel Software and Solutions Group (SSG), Developer Relations Division, Mobile Enterprise Enabling team. He focuses on the security and manageability of enterprise application.

Zhihao Yu is an application engineer in Intel Software and Solutions Group (SSG), Developer Relationship Division, responsible for Intel® TEE technologies enabling and supporting secure payment solution based on Intel® platforms.

Liang Zhang is an application engineer in Intel Software and Solutions Group (SSG), Developer Relationship Division, responsible for supporting enterprise apps and Internet of Things developers based on Intel® platforms.

Link to Part 2

References

¹ TEE System Architecture v1.0: http://www.globalplatform.org/specificationsdevice.asp
² Intel® Software Guard Extensions (Intel® SGX), ISCA 2015 tutorial slides for Intel® SGX: https://software.intel.com/sites/default/files/332680-002.pdf
³ Using Innovative Instructions to Create Trustworthy Software Solutions: https://software.intel.com/en-us/articles/using-innovative-instructions-to-create-trustworthy-software-solutions
⁴ Intel® 64 and IA-32 Architectures Software Developer Manuals: http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html
⁵ Ravi Sahita and Uday Savagaonkar. “Towards a Virtualization-enabled Framework for Information Traceability (VFIT).” In Insider Attack and Cyber Security Volume 39 of the series Advances in Information Security, pp 113-132, Springer, 2008.
⁶ Intel® Identity Protection Technology (Intel® IPT): http://ipt.intel.com/Home
⁷ INTRODUCTION TO INTEL® AES-NI AND INTEL® SECURE KEY INSTRUCTIONS: https://software.intel.com/en-us/node/256280
⁸ Intel® RealSense™ technology: http://www.intel.com/content/www/us/en/architecture-and-technology/realsense-overview.html

By Karen Marcus

Recently, Intel UX Innovation Manager Dr. Daria Loi and her team conducted a study to determine how technology users around the world feel about their computing devices. The study, titled Current Usages: A Global Perspective, explored which devices are people’s favorites, which they use most often, which they use for certain tasks (and why), and what they wish the devices could do better. The research included 1,200 people from six countries across an age range of several decades.

The study found that, while devices such as smartphones, laptops, desktops, and tablets allow users to perform many critical work and personal functions—and that most users have a favorite device (see Figure 1)—there are many areas for improvement. This article discusses the study’s findings in detail and suggests numerous opportunities for developers to create software that can perfect users’ experience.

Figure 1.Study responses for favorite device

No One Size Fits All

Findings from the Current Usages: A Global Perspective study showed that for most people, their smartphone is their favorite device (39%), followed by laptops (30%), and then desktops (21%). But none of these devices can provide the interface to help users achieve every task as they move among work, personal chores, and play. For example, phones are great for taking pictures, listening to music, participating in social media, and communicating with others, but because of screen size and (true or perceived) security reasons, they’re not as useful for tasks like shopping, banking, image editing, and emailing.

Dr. Loi observes, “A smartphone is portable, so that’s the device people tend to reach for most often. Because it’s a favorite device, we want to think it’s perfect, but currently, there’s no perfect device, and that’s why people own multiple devices. However, not all current or emerging users—especially those in developing nations and young users—can afford multiple devices, nor do they want to carry them around. So, as technology developers, the question for us is, ‘How can we create a future that enables people to do everything they want without the bother and expense of multiple devices? How can we help them streamline?’”

In looking for such areas for convergence, developers may want to take note of what users already value in their devices. When asked about the most important features of their devices, survey participants most frequently cited as their number-one feature the operating system, and then performance, screen size, ease of use, and brand—in that order (see Figure 2). Loi notes that ease of use should be of particular interest because it has “huge repercussions for developers as ease of use absolutely needs to be there; otherwise, people won’t buy.”

Figure 2. Study responses for most important features

The Device Shuffle

According to the study, some functions don’t skew strongly in any one direction in terms of which device people prefer when performing them. These functions include reading locally stored books, magazines, and text; browsing photos stored locally or online; live chat; interactive text platforms; video chatting and conferencing; and playing casual games. For these functions, says Loi, “People are moving across devices, depending on the capabilities of each one.”

They may also use different devices based on where they are. When at home, people tend to do things that require a laptop or PC, such as online purchasing, online banking, studying, and playing advanced games. At work or school, they perform tasks on laptops or PCs—including presenting reports, creating and editing documents, emailing, and reading online news or text—and smartphones, including checking the weather and using a calendar. Another location where people tend to use PCs more is Internet cafes and similar dedicated spaces, where typical tasks include using VoIP, video conferencing, and updating blogs or websites.

When on the go, smartphones are the device of choice for tasks like searching locations, navigating, taking pictures or recording videos, and listening to music. “People have expectations based on where they are,” explains Loi. “Their tolerance in terms of responsiveness is different depending on their location. What they expect to see changes when the context changes. Here, the goal for developers is to create applications that work well across contexts or are capable of changing so that the experience for users becomes seamless, no matter what the situation.”

Smartphones Getting Smarter

Whereas some tasks are device-agnostic, others are more commonly performed on a specific device. Forty-three percent of study participants identified smartphones as their device of choice for everyday tasks like checking weather, storing contacts, and using a calendar; 47% said that smartphones were best for searching locations and navigation; and 62% said that smartphones are best for recording photos and video (see Figure 3).

Figure 3.Study responses for favorite device for locate and check functions

Yet, people hesitate to conduct online shopping on their smartphone because they have a sense that it’s less secure. Some participants specifically mentioned that they can’t always see the entire screen and are concerned that they might not see a button or other element they need to be aware of to ensure they’re completing the purchase according to their wishes. However, Loi says, “People would like to perform these tasks on any device, so that’s an opportunity for developers to create the software and infrastructure to enable them to do it safely and securely. This will become increasingly important as Google and Apple push their solutions for using smartphones as credit cards.”

Other potential areas for improvement with smartphones include functions to reduce people’s reliance on other devices when on the go or when those devices are not available. For example, how could a smartphone provide richer productivity capabilities? Considering that people increasingly use their phones as a primary camera, how can camera capabilities be enriched? And, since people want to listen to music on their phone, can smartphones be equipped with better functionality and user interfaces?

Developers have yet another opportunity with smartphones: making them more useful for education. Loi notes, “Teens and young adults cannot imagine living without their phones, yet these devices are rarely integrated into school curricula. In fact, we ask them to leave their devices home or to turn them off; yet, to them, a smartphone is an appealing, familiar, exciting everyday tool that offers an opportunity to make learning stick. In this context, our challenge is to work with educators to develop solutions that enable fluid, seamless, delightful, relevant, deep learning.”

PC Performance Wins

Despite the love affair people have with their phones, when it comes to certain tasks, their laptops, All-in-Ones, and notebooks are their preferred devices (see Figure 4). Specifically, 38% of users choose laptops for editing or modifying media, 47% choose them for creating or editing documents, 36% want them for updating blogs or websites, 41% use them for online banking and bill payment, 44% use them to browse products for potential purchase, and 41% use them to purchase products online. Loi notes that screen size is a primary reason for these preferences, “It’s easier to move around and do fine-grained work on a larger screen.” In addition, she says, “Many software packages aren’t available for smartphones or are prohibitively expensive. The ecosystem of software plus the practical, physical performance advantage of laptops make them the device of choice for these tasks.”

Figure 4.Study responses for favorite device for online purchasing

Survey participants also favored laptops for communication and entertainment functions, such as watching online videos (36%); watching locally stored videos (37%); uploading, downloading, sharing, or transferring media (38%); emailing (42%); and voice over IP (VoIP) and other online voice platforms (35%; see Figure 5). Loi explains, “These are applications that require the high performance that smartphones don’t yet have. In the future, the capabilities of PCs may migrate to smaller devices. Again, the right ecosystem, software, middleware, interface, and infrastructure—in addition to the right ports to easily transfer media—would be needed to make that happen.”

Figure 5. Study responses for favorite device for writing and talking

Other areas where survey participants preferred the performance and screen size of their laptops were presenting reports (59%), playing advanced games (24% for laptops and 25% for desktops), browsing or researching information online (38%), studying (47%), and reading online news or text (39%; see Figure 6)). Loi observes, “People can perform some of these functions from a smartphone, but phones don’t talk to other devices (such as projectors) as well as computers do; their performance doesn’t enable usages such as advanced gaming; and their limited screen size is an issue when engaging in focused, prolonged, and multi-tasking-rich usages such as studying or researching.”

Figure 6.Study responses for favorite device for reading, learning, and research

Speaking of Technology

To better understand participants’ feelings about their devices, Loi and her team interviewed a number of them to gather qualitative information to enrich the quantitative data gathered through the global survey. One important finding was that people increasingly have a love–hate relationship with technology. Loi says, “People realize that they can’t live without their devices, but they also feel that they’re enslaved by them. A typical sentiment was, ‘I want to be able to continuously leverage devices to do the things that are important to me, but I also want them to be smart enough to get out of the way when I don’t need them.’ For example, if someone is in a meeting, his or her smartphone should have this contextual awareness and prevent incoming calls. There’s a lot of potential for creating machines that can learn users’ behavior and take initiatives to disengage when not needed.”

“I miss the times when we weren’t always ON.” —Aron, 20s

On the “love” side of the love–hate spectrum, study participants had things they appreciated about their devices, including that they help to streamline life, connect with others, and interface in real time. But, Loi observes, even the benefits that people like could be better. She queries, “How can we enrich the way we connect with others, beyond existing tools? What technological solutions can be developed to make us feel more present and connected with others, regardless of physical distance?”

“I want to connect to family more; we seem to be looking at [our] devices instead of talking to each other.”
—Carol, 40s

Along with an appreciation of technology comes frustration. Some of the top areas people wish they could change were that device batteries die too fast, that devices slow down or freeze, that they’re susceptible to viruses and privacy intrusions, that they’re too big for true portability, that they’re too small to store everything, and that they’re difficult to operate. A key irritation for study participants was that they want to be more mobile. Loi says, “We’re an increasingly mobile society—a society that fully relies on devices to get things done. But, to be useful tools, those devices need battery life. When I talk to users, it’s clear that charging has become an obsession for most, an impediment to being truly mobile and effective. Among other opportunities, wireless charging can provide great benefits in this context; yet, there needs to be an efficient, reliable, intuitive ecosystem to deliver that usage.” She notes that passwords are another “problem we need to solve,” along with technology changing too quickly, which would be helped through efforts to enable people to transition from older to newer operating systems and applications.

“Stuff changes quickly all the time; it’s hard to keep up.” —Raul, 20s

In response to these frustrations, study participants had thoughts about how the technology could serve them better. They expressed wanting it to be more personalized, power efficient, affordable, ubiquitous, more voice-based, and operating system agnostic, among other things. People also want technology that’s less in the way and more capable of contextually anticipating some user needs. Loi remarks, “People want power, control, and options. Another key factor is ease of use. For developers, usability has to be a top priority. If a user can’t find a function, then it’s essentially not there, and people won’t buy the application.”

“I like to make choices. I don’t want to be told what to do.” —Francis, 30s

Despite the frustration of constant change, people are excited to see what comes next. Loi says that many of the study participants described futuristic features they’d like to see on their devices—what some participants called Iron Man experiences. These features include the ability to think messages to communicate rather than having to speak or type them; live holographic images to chat with; and a super-sensitive microphone that could pick up voice commands from anywhere in the home. Loi says, “Social media and Hollywood give people ideas and expectations for what might be next. Some people don’t understand why these futuristic technologies aren’t already available. Despite their complexity, many expect them to be on the market soon, in many cases sooner than they actually will be .”

“The ability to show someone else exactly what you are imagining. . . like a 3-dimensional hologram.” —Sheila, 20s

Summary

Significant strides have been made in computing technology in recent years, and there are many things people love about their devices. Yet there are many areas in which technology could be even more convenient and user friendly. Dr. Loi of Intel headed the study, Current Usages: A Global Perspective, which set out to understand how people around the world use their devices now and how they would like to use them in the future. The study revealed the following areas of opportunity for developers:

• Streamline functionality so that people can carry (and spend money on) fewer devices.
• Continue to improve usability.
• Improve security and visibility on smartphones for online shopping.
• Improve smartphone functions (such as cameras and music access) to take the place of separate devices.
• Make smartphones more useful for education.
• Make PC software packages more available and useful for smartphones, and give smartphones the capacity to handle more robust applications.
• Create a more seamless, contextual experience for users.
• Improve communications technology to make people feel closer to those with whom they’re communicating.
• Make battery charges last longer, and find new ways to charge devices.
• Help people migrate from older applications and systems to newer ones.

The key, says Loi, is delivering a better vision for the future. For this to happen, she says, “It’s critical for all new developments to be grounded in an understanding of what people do, want, need, and desire. This research is meant to influence what we as an industry do based on a clear, data-driven understanding of what users are doing and what they care about.”

About the Author

Karen Marcus, M.A., is an award-winning technology marketing writer who has 18 years of experience. She has developed case studies, brochures, white papers, data sheets, articles, website copy, video scripts, and other content for such companies as Intel, IBM, Samsung, HP, Amazon Web Services, and Microsoft. Karen is familiar with a variety of current technologies and solutions, such as cloud computing, enterprise computing, personal computing, IT outsourcing, operating systems, and application development.

Installation of Intel® Parallel Studio XE, Intel® System Studio, Intel® INDE, or  Intel® XDK may fail with a message from the installer 'Package signature verification failed.'  There are two specific causes of this failure.  The most likely cause is that valid trusted root certificates cannot be found on the system.  These are needed by the installer to verify the package is good.  A secondary cause is that the installation package is corrupted, or that the package has an invalid signature or timestamp.

If valid trusted root certificates cannot be found on the system, the reasons may include:

• The system either does not have access to the Internet, or if it does, Windows Update cannot acquire the needed certificates.  In either case, the needed certificates will need to be downloaded and installed separately.  Two certificates are needed; one to verify the digital signature, and another to verify the timestamp. 
  • The 'AddTrust External CA Root' certificate is needed to verify the digital signature and may be obtained here.
  • The 'QuoVadis Root Certification Authority' certificate is needed to verify the timestamp and may be obtained here.
• The operating system is not a supported version.  For example, Microsoft* Windows 7* is the oldest version supported for installation of  Intel® Parallel Studio XE 2016; older versions of the OS may not have valid certificates, or the OS may not support valid certificates.  For a complete listing of supported operating systems, see the Intel® C++ Compiler 16.0 Update 1 for Windows* Release Notes (all components of Intel® Parallel Studio XE 2016 have the same OS requirements).
• For the operating system requirements of  Intel® System Studio, Intel® INDE, or  Intel® XDK, see the Release Notes for those products.

If the installation package is corrupted, or the package has an invalid signature or timestamp, then it may have been corrupted during a download, or perhaps been tampered with or corrupted by a virus.  The solution is to obtain a fresh copy of the package and attempt the install again.

< Getting Started with Intel® System Studio for Microcontroller >

 This document provides general overview of the Intel® System Studio for Microcontrollers, information how to use it for developing and debugging your applications for the Intel® Quark^TM microcontroller D1000 on Linux* platforms from the command line and from Eclipse* IDE, gives a list of compiler options and points to additional product information and technical support.

 Intel® Quark^TM microcontroller D1000 requires only mini-usb connection for Flashing, GDB Debugging with OpenOCD connection, and UART communications.

< Introducing the Intel® System Studio for Microcontrollers >

 Intel® System Studio for Microcontrollers is an integrated tool suite for developing and debugging systems and applications for the Intel® Quark™ microcontroller D1000 target - a configurable and fully synthesizable accelerator and microcontroller core (hereinafter is often referred to as “MCU”). Further in this document, we will refer to the Intel® System Studio for Microcontrollers as the “suite”, the “toolchain”, or the “toolset”.
The toolset consists of the following components:

• C/C++ LLVM-based compiler with MCU support including Linker, assembler, C/C++ Run-time libraries
• GDB Debugger with MCU support
• OpenOCD with MCU support

You can use the toolset from the command line and from the Eclipse* Luna, Mars IDE.
The toolset supports the following host operating systems:

• Linux* (Fedora* 19 and Ubuntu* 12.04 LTS and 14.04 LTS)

Installing the Intel® System Studio for Microcontrollers

Download the Intel® System Studio for Microcontrollers from the Intel Registration Center page.
Before installing the toolchain, make sure you have at least 140 Mb free space on your disk.
The name of the archive is:
- l_cembd_iqd_p_1.0.n.xxx.tgz (for Linux*)
where “n” is the “update release” number and “xxx” represents the package build number.

Install the toolchain by extracting the contents of the archive corresponding to your operating system
to a directory where you have the write access. Note that there is no default installation directory for the
toolchain. Make sure the installation directory does not have spaces.

Extract the contents of the archive to a directory where you have the write access, for example, your
$HOME directory. Use the following command:

tar –xzf l_cembd_iqd_p_1.0.0.001.tgz –C $HOME

In this example, your installation directory will be $HOME/l_cembd_iqd_p_1.0.n.xxx.

Installing a valid version of glibc

Make sure you have a valid version of the GNU C Library (glibc). Visit http://www.gnu.org/software/libc/ for installation.

For Fedora* it is glibc.i686. Execute the following command from a terminal as root:

yum install glibc.i686

For Ubuntu* it is ia32-libs. Execute the following command from a terminal as root:

apt-get install ia32-libs

Installing USB Driver

 By default, non-root users do not have access to the JTAG pods connected via USB. You must grant write access to the proper /dev/bus/usb entry every time a device is connected to be able to run OpenOCD using a non-root account.

The process can be automated by adding an udev rule:
1. Create a text file in the rules directory:

sudo vim /etc/udev/rules.d/99-openocd.rules

2. Type the following content:

SUBSYSTEM=="usb", ATTR{idVendor}=="0403", ATTR{idProduct}=="6010",MODE="0666"

3. Unplug the device and plug again (or reboot the system)
Take these steps, otherwise, OpenOCD fails to run with an error message:

Error: libusb_open() failed with LIBUSB_ERROR_ACCESS
Error: no device found
Error: unable to open ftdi device with vid 0403, pid 6010, description '*'
and serial '*'

4. Check what exactly we can see after pluging in D1000 board after successful installation. Type  'sudo dmesg -c'and then plug in the board to your machine and then type 'sudo dmesg -c' once again.

Compiling The Project + Debugging The Project

Please refer the attached PDF user guide for the details.

FirmWare Example

You can modify the firmware that comes with Intel System Studio for Microcontroller package. The below screenshot is a modified version of PushButton test from the firmware.

It detects a button push and prints out a string through the UART.

Reference Number : DPD200570470, DPD200570483

Version : Intel^® C++ Compiler Version 16.0 and earlier; Microsoft* Visual C++* Version 2013 and later

Operating System : Windows*

Problem Description :  An unexpected segmentation fault  or incorrect results may be seen at run-time when applications that access the floating-point environment, for example by making use of the C99 floating-point environment header file fenv.h, are built partly with the Intel C++ compiler and partly with the Microsoft Visual C++ Compiler version 2013 or later.

Cause : There are several differences between the fenv.h header file introduced in the 2013 version of Microsoft Visual C++  and the version provided In the Intel C++ compiler version 16 and earlier. These include a different size for the fenv_t data type, other differences in the definitions for fenv_t and fexcept_t and differences in the definitions of macros, especially FE_DFL_ENV.

Resolution Status : This is a known issue that may be substantially resolved in a future compiler version. However, changes to enhance compatibility of implementations of fenv.h in future Intel compilers with implementations of fenv.h in the Microsoft compiler may lead to incompatibilities with implementations of fenv.h in older Intel compilers.  

Workaround : Build the entire application with the Intel compiler.

This sample uses the Intel® RealSense™ SDK to scan and map a user’s face onto an existing 3D character model. The code is written in C++ and uses DirectX*. The sample requires Intel® RealSense™ SDK R5 or greater, which can be found here.

The sample is available at https://github.com/GameTechDev/FaceMapping.

Scanning

The face scanning module is significantly improved in the WM5 SDK. Improvements include:

• Improved color data
• Improved consistency of the scan by providing hints to direct the user’s face to an ideal starting position
• Face landmark data denoting the positions of key facial features

These improvements enable easier integration into games and other 3D applications by producing more consistent results and requiring less user modification.

The scanning implementation in this sample guides the user’s head to a correct position by using the hints provided from the Intel® RealSense™ SDK. Once the positioning requirements are met, the sample enables the start scan button.

This sample focuses on the face mapping process, and therefore the GUI for directing the user during the scan process is not ideal. The interface for an end-user application should better direct the user to the correct starting position as well as provide instructions once the scan begins.

The output of the scan is an .OBJ model file and an associate texture which will be consumed in the face mapping phase of the sample.

Figure 1: The face scanning module provides a preview image that helps the user maximize scan coverage.

Figure 2: Resulting scanned mesh. The image on the far right shows landmark data. Note that the scan is only the face and is not the entire head. The color data is captured from the first frame of the scan and is projected onto the face mesh; this approach yields high color quality but results in texture stretching on the sides of the head.

Face Mapping

The second part of the sample consumes the user’s scanned face color and geometry data and blends it onto an existing head model. The challenge is to create a complete head from the scanned face. This technique displaces the geometry of an existing head model as opposed to stitching the scanned face mesh onto the head model. The shader performs vertex displacement and color blending between the head and face meshes. This blending can be performed every time the head model is rendered, or a single time by caching the results. This sample supports both approaches.

The high-level process of this mapping technique includes:

1. Render the scanned face mesh using an orthographic projection matrix to create a displacement map and a color map.
2. Create a matrix to project positions on the head model onto the generated displacement and color maps. This projection matrix accounts for scaling and translations determined by face landmark data.
3. Render the head model using the projection matrix to map vertex positions to texture coordinates on the displacement and color maps.
4. Sample the generated maps to deform the vertices and color the pixels. The blending between the color map and the original head texture is controlled by an artist-created control map.
5. (Optional) Use the same displacement and blending methodologies to create a displaced mesh and single diffuse color texture that incorporates all blending effects.

Art Assets

The following art assets are used in this sample:

Head model. A head model the face is applied to. The model benefits from higher resolution in the facial area where the vertices are displaced.

Feature map. Texture mapped to the UVs of the head model that affects the brightness of the head.

Detail map. Repeated texture that applies additional detail to the feature map.

Color transfer map. Controls blending between two base skin tones. This allows different tones to be applied at different locations of the head. For example, the cheeks and ears can have a slightly different color than the rest of the face.

Control map. Controls blending between the displacement and color maps and existing head model data. Each channel of the control map has a separate purpose:

• The red channel is the weight for vertex Z displacement. A weight of zero uses the vertex position of the head model, a weight of one modifies the Z vertex position based on the generated displacement map, and intermediate values result in a combination of the two.
• The green channel is the weight for blending between the head diffuse color and the generated face color map. Zero is full head diffuse and one is full face color.
• The blue channel is an optional channel that covers the jawbone. This can be used in conjunction with the green channel to allow a user’s jawbone color to be applied instead of the head model’s diffuse color. This might be useful in the case where the user has facial hair.

All maps are created in head model UV space.

Figure 3: Head model asset with a highly tesselated face. The scanned face will be extruded from the high resolution area.

Figure 4: Feature map (left) and detail map (right). The detail map is mapped with head model UVs but repeated several times to add detail.

Figure 5: Color transfer map (left) and the color transfer map being applied drawn on the head model (right). This map determines the weights of the two user-selected skin colors.

Figure 6: The control map (left) and the control map applied to the head model. The red channel is the area that should be affected by the displacement map. The green channel is the area that should receive the scanned color map. The blue channel represents the jawbone area. Using the color map in the jawbone area can be used to capture distinct jawbone features such as facial hair.

Displacement and Color Maps

The first step of the face mapping process is generating a displacement and color map based on the scanned face mesh. These maps are generated by rendering the face mesh using an orthographic projections matrix. This sample uses multiple render targets to generate the depth displacement map and the color map in a single draw call. It sets the projection matrix so that the face is fully contained within the viewport.

Figure 7: The displacement and color maps generated from the scanned face mesh.

Map Projection Matrix

Now we use the landmark data from the scanned face model and the head model to create a transformation matrix to convert from head model vertex coordinates in model space to texture coordinates in the displacement and color map space. We’ll call this the map projection matrix because it effectively projects the displacement maps onto the head model.

The map projection matrix consists of a translation and a scale transformation:

• Scale transform. The scaling factor is calculated by the ratio of the distances between the eyes of the scanned face mesh (in projected map coordinates) and eyes of the head model.
• Translation transform. The vertex translation is calculated using the head model and scanned face mesh landmark data. The translation makes the point directly between the eyes of the head model to align with the respective point on the displacement map. To calculate this respective point, we use the left and right eye landmarks to calculate the center point and then transform it by the orthographic projection matrix used when generating the displacement and color maps.
• Rotation transform. This sample assumes that the scanned face mesh is axially aligned and does not require a rotation. The sample includes GUI controls for introducing rotation for artistic control.

Figure 8: Generated color map (left) being orthographically projected onto the head model. The map is translated and scaled so that the yellow anchor points between the eyes align.

Rendering

The sample applies the generated displacement and color maps at render time in vertex and pixel shaders.

Vertex Shader

The vertex shader displaces the model’s Z coordinate based on the displacement map. The displacement map texture coordinates are sent to the pixel shader where they’re used to sample the color map for blending.

The vertex shader steps include:

1. Transform vertex position by the map projection matrix to get color/displacement map texture coordinates.
2. Sample the displacement map texture at calculated coordinates.
3. Convert the displacement sample to a model space Z value. The range and scale of the displacement map are passed through a constant buffer.
4. Blend between the displaced Z value and the original Z value based on the control map’s red component. The control map lets the artist decide what vertices get displaced and allows for a gradual, smooth transition to the displaced position.
5. Pass the displacement map UV coordinates to the pixel shader to be used to sample the color map.

Pixel Shader

The pixel shader uses the control map’s green channel to blend between the head color and the generated color map texture. Because the sample allows the user to change the head color to better match the scanned face color, the color is blended into greyscale art assets in the pixel shader. The skin color is calculated for each pixel by blending between two user-selected colors based on the color transfer map. That skin color is multiplied by the greyscale intensity to produce a final head color.

Figure 9: Demonstration of head model blending without applying the displacement or color maps.

Figure 10: Final result composited in real-time using the sample.

Exporting

This technique applies several layers of blending inside the pixel shader as well as modifies the vertex position in the vertex shader each time the model is rendered. The sample also supports exporting the composited texture and the deformed mesh to an .OBJ file.

The entire compositing and deformation process still occurs on the GPU using a variation of the original shaders. The new vertex shader uses Direct3D* stream-output support to capture the deformed vertices. The vertex shader also uses the input texture coordinates as the output position; this effectively renders a new UV mapped texture.

Once composited, the model can be rendered with lower overhead and without any custom shaders, allowing it to easily be loaded by 3D modeling tools and game engines.

Figure 11: Exported .OBJ model (left), composited head texture UV mapped to the head model’s UVs.

Introduction

Intel® RealSense™ technology supports two varieties of depth cameras: the short-range, user-facing camera (called the F200) is designed for use in laptops, Ultrabook™ devices, 2 in 1 devices, and All-In-One (AIO) form factors, while the long-range, world-facing camera (called the R200) is designed for the detachable and tablet form factors. Both these cameras are available as peripherals and are built into devices we see in the market today. When using Intel RealSense technology to develop apps for these devices, keep in mind that the design paradigm for interacting with 3D apps without tactile feedback is considerably different than what developers and end users are used to with apps built for touch.

In this article, we highlight some of the most common UX ideas and challenges for both the F200 and R200 cameras and demonstrate how developers can build in visual feedback through the Intel® RealSense™ SDK APIs.

F200 UX and API Guidelines

Outcome 1: Understanding the capture volumes and interaction zones for laptop and AIO form factors

The UX scenario

Consider the scenarios depicted in Figure 1

Figure 1: Capture volumes.

The pyramid drawn out of the camera represents what is called the capture volume, also known as the Field of View (FOV). For the F200, the capture volume includes the horizontal and vertical axes of the camera as well as the effective distance of the user from the camera. If the user moves out of this pyramid, the camera fails to track the mode of interaction. A table of reference for FOV parameters is given below:

Both the color and depth cameras within the F200 have different fidelities, and, therefore, application developers need to keep the capture volume in mind for the modalities they want to use. As shown in the table above, the effective range for gestures is shorter, whereas face tracking covers a longer range.

Why is this important from a UX perspective? End users are unaware of how the camera sees them. Since they are aware of the interaction zones, they may become frustrated using the app because there is no way to determine what went wrong. As shown in the image on the left in Figure 1, the user’s hand is within the FOV, whereas in the image on the right, the user’s hand is outside the FOV, depicting a scenario where tracking could be lost. The problem is compounded if the application uses two hands or multiple modalities like hands and the face at the same time. Consider the consequences if your application is deployed on different form factors like laptops and AIOs where the effective interaction zone in the latter is higher than on a laptop. Figure 2 depicts scenarios where users are positioned in front of different devices.

Figure 2: FOV and form factor considerations.

Keeping these parameters in mind will help you build an effective visual feedback mechanism into the application that can clearly steer users in the right track of usage. Let’s now see how to capture some of these FOV parameters in your app through the SDK.

The technical implementation

The Intel RealSense SDK provides APIs that allow you to capture the FOV and camera range. The APIs QueryColorFieldOfView and QueryDepthFieldOfView are both provided as device-neutral functions within the “device” interface. Here is how to implement it in your code:

Though the return data structure is a PXCPointF32, the values returned indicate the x and y angles in degrees and are the model set values, not the device-calibrated values.

The next parameter with the capture volume is range. The QueryDepthSensorRange API returns the range value in mm. Again, this is a model default value and not the device-calibrated value.

Knowing the APIs that exist and how to implement them in your code, you can build effective visual feedback to your end users. Figures 3 and 4 show examples of visual feedback for capture volumes.

Figure 3: Distance prompts.

Figure 4: World diagrams.

Simple prompts indicate the near and far boundaries of the interaction zone. Without prompts, if the system becomes unresponsive, the user might not understand what to do next. Filter the distance data and show the prompt after a slight delay. Also ensure that you use positive instructions instead of error alerts. World diagrams orient the user and introduce them to the notion of a depth camera with an interaction zone. The use of world diagrams is recommended for help screens and tutorials and for games in which users might be new to the camera. For maximum effectiveness, show the world diagrams only during a tutorial or on a help screen. Instructions should be easy to understand and created with the audience in mind.

You can supplement the use of the above-mentioned APIs with alerts that are provided within each SDK middleware to capture specific user actions. For example, let’s take a look at the face detection middleware. The following table summarizes some of the alerts within the PXC[M]FaceData module:

As we already know, the SDK allows for detecting up to four faces within the FOV. Using the face ID, we can capture alerts specific to each face depending on your application’s needs. It is also possible that tracking is lost completely (example: The face moved in and out of the FOV too fast for the camera to track). In such a scenario, you can use the capture volume data together with the alerts to build a robust feedback mechanism for your end users.

The SDK also allows you to detect occlusion scenarios. Please refer to the F200 UX guideline document for partially supported and unsupported scenarios. Irrespective of which category of occlusion you are trying to track, the following set of alerts will come in handy.

Now let’s take a look at alerts within the hand tracking module. These are available within the PXC[M]HandData module of the SDK. As you can see, some of these alerts also provide the range detection implicitly (recall that the range is different for the face and hand modules).

Now that you know what capabilities the SDK provides, it is easy to code this in your app. The following code snippet shows an example:

Replace the wprintf_s statements with logic to implement the visual feedback. Instead of enabling all alerts, you can also just enable specific alerts as shown below:

Figures 5 and 6 show examples of effective visual feedback using alerts.

Figure 5: User viewport.

Figure 6: User overlay.

Links to APIs in SDK documentation:

QueryColorFieldOfView: https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?querycolorfieldofview_device_pxccapture.html

QueryDepthFieldOfView: https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?querydepthfieldofview_device_pxccapture.html

QueryDepthSensorRange: https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?querydepthsensorrange_device_pxccapture.html

Face module FOV alerts:https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?alerttype_alertdata_pxcfacedata.html

Hand module FOV alerts:https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?manuals_handling_alerts.html

Outcome 2: Minimizing user fatigue

The UX scenario: The choice of appropriate input for required precision

When building apps using the Intel RealSense SDK, it is important to keep modality usage relevant. Choosing the appropriate input methods for various scenarios in your application plays a key role. Keyboard, mouse, and touch provide for higher precision, while gesture provides lower precision. For example, keyboard and mouse, rather than gestures, are still the preferred input methods for data-intensive apps. Imagine using your finger instead of a mouse to select a specific cell in Excel (see Figure 7). This would be incredibly frustrating and tiring. Users naturally tense their muscles when trying to perform precise actions, which in turn accelerates fatigue. 

Figure 7: Choice of correct input. 

The selection of menu items can be handled either through touch or mouse. The Intel RealSense SDK modalities provide for a direct, natural, and non-tactile interaction mechanism while making your application more engaging. Use them in a way that does not require many repeating gestures. Continuous and low risk actions are best for gesture usage.

Choice of direction for gesture movement

Tips for designing for left-right or arced gesture movements: Whenever presented with a choice, design for movement in the left-right directions versus up-down for ease and ergonomic considerations. Also, avoid actions that require your users to lift their hands above the height of their shoulder. Remember the gorilla arm effect?

Figure 8: Choice of direction for gesture movement.

Choice of relative versus absolute motion

Allow for relative motion instead of absolute motion wherever it makes sense. Relative motion allows the user to reset his or her hand representation on the screen to a location that is more comfortable for the hand (such as when lifting a mouse and repositioning it so that it is still on the mouse pad). Absolute motion preserves spatial relationships. Applications should use the motion model that makes the most sense for the particular context.

Understanding speed of motion

The problem of precision is compounded by speed. When users move too fast in front of the camera, they potentially risk losing tracking altogether because they could move out of the capture volume. Building fast movement into apps also introduces fatigue while being more error prone. So it is critical to understand the effects of speed and its relation to the effective range (faster motion up to 2m/s can be detected closer to the camera—20 to 55 cm) and the capture volume (closer to the camera implies only one hand can be in the FOV).

Understanding action and object interaction

The human body is prone to jitters that could be interpreted by the camera as multiple interactions. When designing apps for the Intel RealSense SDK, keep action-to-object interaction in mind. For example, if you have objects that could be grabbed through gesture, consider their size, placement, how close they are to the edges of the screen, where to drop the object, and how to detect tracking failures.

Here are some guidelines to help avoid these challenges:

• Objects should be large enough to account for slight hand jitter. They should also be positioned far enough apart so users cannot inadvertently grab the wrong object.
• Avoid placing interaction elements too close to the edge of the screen, so the user doesn’t get frustrated with popping out of the field of view and thus lose tracking altogether.
• If the interface relies heavily on grabbing and moving, it should be obvious to the user where a grabbed object can be dropped.
• If the hand tracking fails while the user is moving an object, the moved object should reset to its origin and the tracking failure should be communicated to the user.

The technical implementation: Speed and precision

If your application doesn’t require the hand skeleton data, but relies more on quicker hand movements, consider using the “blob” module. The following table gives a sampling of scenarios and their expected precision. While full hand tracking with joint data requires a slower speed of movement, this limitation can be overcome by either choosing the extremities or the blob mode. The blob mode is also advantageous if your application is designed for kids to use.

If you do want more control within your app and want to manage the speed of motion, you can obtain speed at the hand joint level through the use of PXCMHandConfiguration.EnableJointSpeed. This allows you to either obtain the absolute speed based on current and previous location or average speed over time. However, this feature is a drain on the CPU and memory resources and should be considered only when absolutely necessary.

Since hand jitters cannot be avoided, the SDK also provides the Smoother utility (PXC[M]Smoother) to reduce the numbers of jitters as seen by the camera. This utility uses various linear and quadratic algorithms that you can experiment with based on your needs and pick the one that works best. In Figure 9 below, you can see how the effect of jitters is reduced through the use of this utility.

Figure 9: Smoothed and unsmoothed data.

Another mechanism you can use to detect whether the hand is moving too fast is the TRACKINGSTATUS_HIGH_SPEED enumeration within the PXCMHandData.TrackingStatusType property. For face detection, fast movements may lead to lost tracking. Use PXCMFaceData.AlertData.AlertType – ALERT_FACE_LOST to determine whether tracking is lost. Alternatively, if you are using hand gestures to control the OS using Touchless Controller, use the  PXC[M]TouchlessController member functions SetPointerSensitivity and SetScrollSensitivity to set pointer and scroll sensitivity.

Bounding boxes

An effective mechanism to ensure smooth action and object interactions is the use of bounding boxes, which help provide clear visual cues to the user on the source and destination areas for object of interaction.

The hand and face modules within the SDK provide for the PXCMHandData.IHand.QueryBoundingBoxImage API, which returns the location and dimension of the tracked hand—a 2D bounding box—in the depth image pixels, and the PXCMFaceData.DetectionData.QueryBoundingRect API, which returns the bounding box of the detected face. You can also use PXCMHandData.AlertType – ALERT_HAND_OUT_OF_BORDERS to detect whether the hand is out of the bounding box.

Links to APIs in the SDK documentation:

Blob tracking algorithm:https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?manuals_blob_tracking.html

EnableJointSpeed:https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?enablejointspeed_pxchandconfiguration.html

The Smoother utility:https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?manuals_the_smoother_utility.html

TouchlessController:https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?pxctouchlesscontroller.html

SetPointerSensitivity and SetScrollSensitivity:https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?member_functions_pxctouchlesscontroller.html

R200 UX and API Guidelines

The R200 camera is designed for tablet and detachable form factors with uses that capture the scene around you. Augmented reality and full body scan are some of the prominent use cases with the R200 camera. With the focus on the world around you, the nature and scope of UX challenges is different from the F200 scenarios we discussed in the previous section. In this section, we provide insights into some of the known UX issues around the Scene Perception module (which developers will use for augmented reality apps) and the 3D scanning module.

Outcome 1: Understanding the capture volumes and interaction zones for tablet form factors

The UX scenario

As shown in Figure 10, the horizontal and vertical angles and the range for the R200 are considerably different than for the F200. The R200 camera can also be used in two different modes: active mode (when the user is moving around capturing a scene) and passive mode (when the user is working with a static image). When capturing an object/scene, ensure that it is within the FOV while the user is actively performing a scan. Also note how the range of the camera (depending on indoor versus outdoor use) is different compared to the F200. How do we capture these data points in runtime, so that we can provide good visual feedback to the user?

Figure 10: R200 capture volumes.

The technical implementation

The QueryColorFieldOfView() and the QueryDepthFieldOfView() APIs were introduced in the F200 section above. These functions are device neutral and will work to capture the capture volumes for R200 as well. However, the API to detect the R200 camera range is device specific. To obtain this data for the R200, you must use the QueryDSMinMaxZ API, which is available as part of the PXCCapture interface and returns the minimum and maximum range of the camera in mm.

Links to APIs in SDK documentation

QueryDSMinMaxZ: https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?querydsminmaxz_device_pxccapture.html

Outcome 2: Understanding user action and scene interaction

The UX scenario: Planning for the scene and camera qualities

While working in the active camera mode, be aware of the camera limitations. Depth data is less accurate when scanning a scene with very bright areas, reflective surfaces, and black surfaces. Knowing when tracking could fail helps build an effective feedback mechanism into the application and fail the app gracefully than preventing play.

The technical implementation

The Scene Perception and 3D scanning modules have different requirements and hence provide for separate mechanisms to detect minimum requirements.

• Scene Perception. Always use the CheckSceneQuality API within the PXCScenePerception module to tell whether the scene in question is suitable for tracking. The API returns a value between 0 and 1. The higher the return value, the better the scene is for tracking. Here is how to implement it in the code:

Once you determine that the scene quality is adequate and tracking starts, check the tracking status dynamically using the TrackingAccuracy API within the PXCScenePerception module, which enumerates the tracking accuracy definition.

To ensure the right quality of data for the scene in question, you can also set the voxel resolution (a voxel represents the unit/resolution of the volume). Depending on whether you are tracking a room-size area, tabletop, or a close object, for best results, set the voxel resolution as indicated in the table below.

• 3D Scanning. The 3D scanning algorithm provides the alerts shown in the table below. Use PXC3DScan::AlertEvent to obtain this data.

Once the data to track camera and module limitations is available within the app, you can then use that to provide the visual feedback, clearly demonstrating to users how their actions were translated by the camera or in the event of failure, showing them how they can correct their actions for better usage. Samples of visual feedback are provided here for reference; you can adapt these to suit the application requirement and UI design.

• Sample tutorial at the start:
  Figure 11: Tutorials.
   
• Preview of subject or area captured:
  
  Figure 12: Previews.
   
• User prompts:
  
  Figure 13: User prompts.

Minimizing fatigue while holding the device

Most applications will use the device in both active and inactive camera modes. (We distinguish these two modes as follows: “active camera” when the user is holding up the tablet to actively view a scene through the camera or perform scanning and “inactive camera” when the user is resting the tablet and interacting with content on the screen while the camera is off.) Understanding the way in which the user holds and uses the device in each mode and choosing interaction zones accordingly is critical to reducing fatigue. Active camera mode is prone to a higher degree of fatigue due to constant tracking, as shown in Figure 14.

Figure 14: Device usage in active and inactive modes.

Choosing the appropriate mode for the activity

The mode of use also directly dictates the nature of interaction with the app you build through the UI. In active mode, the user uses both hands to hold the device. Therefore, any visual elements, like buttons that you provide in the app, must to be easily accessible to the user. Research has shown that the edges of the screen are most suitable for UI design. Figure 15 shows the touch zones that are preferred. The interactions are also less precise in active mode, so the active mode works best for short captures.

In contrast, in inactive mode, touch interactions are more comfortable, more precise, and can be used for extended play.

Figure 15: Touch zones in active and inactive modes.

Links to APIs in SDK documentation:

Scene Perception Configuration and tracking data:https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?manuals_configuration_and_tra2.html

3D Scanning Alerts:https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?alertevent_pxc3dscan.html

Summary:

App development using the Intel® RealSense™ Technology requires developers to keep end user in mind from the very beginning stages of development. The directions provided in this article provides the starting point for some of the critical UX challenges and implementing them in code using the SDK.

Additional Resources:

F200 UX Guidelines: https://software.intel.com/sites/default/files/managed/27/50/Intel%20RealSense%20SDK%20Design%20Guidelines%20F200%20v2.pdf

R200 UX Guidelines:

https://software.intel.com/sites/default/files/managed/d5/41/Intel%20RealSense%20SDK%20Design%20Guidelines%20R200%20v1_1.pdf

Best UX practices for F200 apps:

https://software.intel.com/en-us/articles/best-ux-practices-for-intel-realsense-camera-f200-applications

Link to presentation and recording at IDF:

http://myeventagenda.com/sessions/0B9F4191-1C29-408A-8B61-65D7520025A8/7/5

About the authors:

Meghana Rao:

As a Developer Evangelist within Intel's Software and Services division, Meghana works with developers and ISVs assisting them with Intel® RealSense™ Technology and Windows* 8 application development on Ultrabook™, 2-in-1s, and tablet devices on Intel® Architecture. She is also a regular speaker at Intel Application Labs teaching app design and development on Intel platforms and has contributed many white papers on the Intel® Developer Zone. She holds a Bachelor of Engineering degree in Computer Science and engineering and a Master of Science degree in Engineering and Technology Management. Prior to joining Intel in 2011, she was a senior software engineer with Infineon Technologies India Pvt. Ltd.

Kevin Arthur:

Kevin Arthur is a Senior User Experience Researcher in the Perceptual Computing group at Intel. He leads user research on new use cases and best practices for RealSense 3D depth cameras, including mixed and augmented reality experiences. He holds a PhD in computer science from the University of North Carolina at Chapel Hill, where he specialized in human-computer interaction in virtual reality systems. He previously did user experience and software R&D work at Amazon Lab126, Synaptics, and Industrial Light & Magic.

Figure 1. Windows* 10 provides a common OS across the spectrum of Microsoft devices.

Among a long list of improved features, Windows 10 is positioned to tie together the wide variety of potential gaming platforms with UWP apps. The augmented support for 2-in-1s brings a more natural adaptability to interaction modes. With all platforms running variants of the same system, the apps are delivered by the same store. For other PC games, the support provided for Steam* makes it even stronger than before. Whichever distribution works for your game, DirectX* 12 brings new life to games with significantly increased power.

Power with DirectX 12

DirectX 12 is the most recent version of Microsoft’s graphics API suite designed to manage and optimize multimedia functionality – most notably handling video game graphics; in fact, Xbox* got its name from originally being conceived as the DirectX Box, so it’s no surprise that the technology has continued to be a pillar of game development.

Figure 2. DirectX* 12 brings PCs console-level optimization

DirectX 12 on Windows 10 improves that relationship further by reducing call overhead, memory paging, and system footprint, to give your game more space and processor time. Games being run in the foreground are granted better control over their process execution, with reserved memory to minimize texture swaps and other potential conflicts with external apps. Without going into a deeper explanation, these capabilities translate into a smoother experience for the user and a wider range of computers being able to maintain acceptable performance.

One of the biggest concerns when considering use of a new technology is how much of the existing content and development process will need to change. Fortunately, the jump from DirectX 11 to 12 keeps interdepartmental workings consistent; development can continue unimpeded with the same textures and models, formats, render targets, compute models, etc. The most significant changes are constrained to the code of custom engines, and MSDN even offers some official guidance for porting those. In most situations, developers use an existing engine—DirectX 12 has already been introduced for Unreal* and Unity* (specifically providing boosts for multicore scaling, with continued improvement ahead). Combined with its native support on 6th generation Intel® Core™ processors, DirectX 12 really gets the engine running on all cylinders.

Distribution with Steam

Steam is the leading digital distribution service for PC games, making it easy for gamers to discover, purchase, and access their games. It may seem counterproductive for Microsoft to support an external app store, but it’s not trying to compete for the same space; the Windows app store is focused on serving the full spectrum of Windows devices (which last year total upward of 1.5 billion), so you can do things like shopping on your phone to find and download games to your Xbox One and PC.

Just like developers moving to new tools, gamers can be hesitant to upgrade to a new OS when so much of their game library is in a potentially unsupported program. Fortunately, Steam* on Windows 10 is fully backward compatible, requiring no changes aside from updated video drivers.

There is even a Steam Tile to create tiles for quick access to individual games. Microsoft really wants Steam to “run great on Windows 10,” and these various points of solid support are definitely gaining traction; according to Steam’s opt-in hardware survey, Windows 10 has become the second most widely used OS on Steam, with over a quarter of users having made the switch in the short few months since its release.

Figure 3. Second only to Windows* 7 (total 44.86%), Windows 10 continues to grow (27.42%) as much as all others combined.

Versatility with 2-in-1s

With the hardware of modern convertible devices, a tablet ideal for gaming on the go can become a laptop with the high-quality graphics and performance expected of PC games. In combination, the versatility afforded to gamers having a touch screen and laptop can create a gestalt experience tailored to the ideal interaction methods for various activities. And with technology like the Intel® RealSense™ SDK enabling perceptual computing with 3D cameras and voice commands (here’s a quick-start guide to using it in Windows 10), the spectrum of user controls continues to grow.

Windows 10 bridges the gap between these two modes, allowing seamless transitions between interfaces tailored for the control scheme currently available to the user; when in tablet mode, most input is likely to be designed for touch, whereas keyboard usage can be expected when the keyboard is present.

Integration across the Windows Universe

The dream goal of “write once, run anywhere” becomes clearer with the advent of Universal Windows Platform (UWP) apps. Microsoft is standardizing the operating systems of all platforms under the Windows 10 banner, creating a significant opportunity for code reuse. While you still have the option of writing a different version for each platform, you can write a single game that runs on a PC, Xbox One, tablets, phones, and even the HoloLens*, or target entire device families by their hardware commonalities.

UWP apps bridge the gaps while allowing device-specific customization. Detecting the device family and specific features available can be done before installation by deciding which binary to use, or it can be done by API contracts at runtime, allowing adaptive features to make things like 2-in-1s and peripherals easier to use.

One notable addition to make the most of Windows Apps is the way app life cycle transitions are handled. Rather than being in the dark as to whether your game is in stasis or even about to be closed, these changes can trigger program events to handle operations for a more persistent player experience. In addition to apps being able to gracefully manage state changes, Windows is able to more effectively manage the resources needed.

There are a number of resources that provide technical information, such as Microsoft’s GDC 2015 talks, presenting good overviews touching on many aspects this article doesn’t have the space to explore: Chris Tector and Don Box explain how the systems work together under this paradigm, Bill Schiefelbein demonstrates how gamers and game developers connect in a new form of social network around the Xbox app, Vijay Gajjala and Brian Tyler elaborate on using the Xbox Live APIs to quickly make use of these new features, and Chris Charla introduces the ID@XBOX program for independent developers to self-publish at the same level as anyone else (even receiving two dev kits at no charge as long as you have an email address with your company website).

Connection via the Xbox App

The Xbox app—the PC hub of gaming activity that extends Xbox Live functionality to other devices—ties all the game experiences together. It unites gamer friends in more of a social network dynamic, driving discovery, engagement, and player retention by enabling development of a player culture.

Figure 4. The Xbox* app is geared toward enriching the gamer experience.

Players can capture clips and screenshots of their games to share, with the bonus option of capturing the last 30 seconds of gameplay for when something awesome but unexpected happens. Since the network of friends and followers connects across platform differences, any game enabled with these features is granted the same degree of exposure—even users viewing the app on a mobile device can watch your game clips.

The single sign-on approach of Xbox accounts makes user profile association easy, letting the OS handle a lot of the leg work (and import friends from Facebook*). Similarly, since Windows apps have explicit manifest information, the system can manage installation and updates, saving significant developer hours (which are especially critical once the game goes live).

Those developers can also simplify gathering and using in-game metrics, granting a richer online presence; rather than simply knowing what game you’re playing, your friends could potentially see where you are in the game and how you’re doing—presumably with privacy options. The friends list is viewable alongside the games library, where developers can live-update the game’s information for announcements and updates.

The Windows Dev Center also provides dashboards on analytics, tracking things like player data and monetization, as well as dynamic creation of engagement drivers like achievements, challenges, and leaderboards. With the information available to developers, players can connect with new aspects of your game, while connecting with others through your game in new ways.

Bonus: Streaming Xbox One to the PC

In addition to its ability to play and connect with friends across platform differences, the Xbox One can stream to any Windows 10 PC on the same network. Granted, there are some considerations to ensure the gameplay stays in top shape, but the ability to play console games without being tethered to a TV seems like a dream come true.

Figure 5. Streaming gameplay to anywhere on your network.

Looks like a Win-Win

If you are a gamer or game developer targeting the PC or Xbox One, Windows 10 is your friend. The features delivered by the Xbox app and APIs lay the foundation for a broader and deeper engagement with players. By anticipating gamer preferences for power and ease of use, the details of Microsoft’s newest offering all lean toward augmenting the gamer experience on multiple levels.

For More Information

The meaning of Xbox: http://www.economist.com/node/5214861

DirectX 12: http://blogs.msdn.com/b/directx/archive/2014/03/20/directx-12.aspx

What DirectX 12 means for gamers and developers: http://www.pcgamer.com/what-directx-12-means-for-gamers-and-developers/

Important Changes from Direct 3D 11 to Direct 3D 12: https://msdn.microsoft.com/en-us/library/windows/desktop/dn899194(v=vs.85).aspx

Porting from Direct3D 11 to Direct 3D 12: https://msdn.microsoft.com/en-us/library/windows/desktop/mt431709(v=vs.85).aspx

Any developer can now make a DirectX 12 game with updated Unreal Engine 4: http://www.windowscentral.com/any-developer-can-now-make-directx-12-game-updated-unreal-engine-4

Unity Founder: DirectX 12 API Alone Doesn’t Give A Significant Performance Boost: http://gamingbolt.com/unity-founder-directx-12-api-alone-doesnt-give-a-significant-performance-boost

Product Brief: 6th Gen Intel® Core™ Processor Platform:  http://www.intel.com/content/www/us/en/processors/core/6th-gen-core-family-mobile-brief.html

Valve Lines Up Console Partners in Challenge to Microsoft, Sony: http://www.bloomberg.com/news/articles/2013-11-04/valve-lines-up-console-partners-in-challenge-to-microsoft-sony

Microsoft’s Xbox Store isn’t trying to cut out Steam in Windows 10: http://venturebeat.com/2015/05/21/microsofts-xbox-store-isnt-trying-to-cut-out-steam-in-windows-10/

Microsoft: We have 1.5 billion Windows devices in the market: http://www.neowin.net/news/microsoft-we-have-15-billion-windows-devices-in-the-market

Steam Tile:   https://www.microsoft.com/en-us/store/apps/steam-tile/9wzdncrfhzkv

Microsoft wants to support Steam and “help it run great on Windows 10”: http://www.technobuffalo.com/2015/08/07/microsoft-wants-to-support-steam-and-help-it-run-great-on-windows-10/

Steam Hardware & Software Survey: November 2015: http://store.steampowered.com/hwsurvey

Intel RealSense: http://www.intel.com/content/www/us/en/architecture-and-technology/realsense-overview.html

Get Started Developing Intel® RealSense™ SDK for Windows* 10 Desktop Apps: https://software.intel.com/en-us/articles/get-started-developing-intel-realsense-sdk-for-windows-10-desktop-apps

Windows 10 on the Surface Pro 3: Now the 2-in-1 makes perfect sense: http://www.gizmag.com/windows-10-surface-pro-3-review/38189/

What's a Universal Windows Platform (UWP) app?: https://msdn.microsoft.com/en-us/library/windows/apps/dn726767.aspx

Dynamically detecting features with API contracts (10 by 10): https://blogs.windows.com/buildingapps/2015/09/15/dynamically-detecting-features-with-api-contracts-10-by-10/

It's Universal: Understanding the Lifecycle of a Windows 10 Application: https://visualstudiomagazine.com/articles/2015/09/01/its-universal.aspx

MSDN Channel 9 – GDC 2015: https://channel9.msdn.com/Events/GDC/GDC-2015

Developing Games for Windows 10: https://channel9.msdn.com/Events/GDC/GDC-2015/Developing-Games-for-Windows-10

Gaming Consumer Experience on Windows 10: https://channel9.msdn.com/Events/GDC/GDC-2015/Gaming-Consumer-Experience-on-Windows-10

Developing with Xbox Live for Windows 10: https://channel9.msdn.com/Events/GDC/GDC-2015/Developing-with-Xbox-Live-for-Windows-10?ocid=SessionsInEvent

New Opportunities for Independent Developers: https://channel9.msdn.com/Events/GDC/GDC-2015/New-Opportunities-for-Independent-Developers?ocid=SessionsInEvent

The Xbox Experience on Windows 10: http://www.xbox.com/en-US/windows-10/xbox-app

Windows Dev Center: https://dev.windows.com/en-us/games

How to use game streaming in the Xbox app on Windows 10: https://support.xbox.com/en-US/xbox-on-windows/gaming-on-windows/how-to-use-game-streaming

About the Author

Brad Hill is a software engineer at Intel in the Developer Relations Division. Brad investigates new technologies on Intel® hardware and shares the best methods with software developers via the Intel® Developer Zone and at developer conferences. His focus is on turning students and developers into game developers and helping them change the world.

Introduction

The much anticipated Intel® RealSense™ SDK support for developing Universal Windows Platform (UWP) apps has arrived in SDK R5 (v7), and this article will help get you started. A sneak peek of the UWP interfaces and methods was presented in the R4 release documentation, and R5 now delivers the software components, samples, and documentation you’ll need for developing UWP apps that use the Intel® RealSense™ camera (SR300).

As stated in the What's News in R5 of the Intel® RealSense™ SDK R5 (v7) article, the SR300 camera should be available for order in Q1 of 2016 and integrated into select Intel-based systems in 2016. Because the SR300 is not yet available for end users, this article will focus on the things you need to know now to prepare for its arrival.

What You’ll Need to Get Started

• Laptop or PC running Microsoft Window 10 RTM
• Microsoft Visual Studio* 2015 with Microsoft Windows 10 Developer Tools
• Intel RealSense camera (SR300) (available for order in Q1 of 2016)
• Intel RealSense SDK R5 (v7)
• SR300 DCM 3.0 Gold (only available through Windows Update)

SDK Scope and Limitations

• The SDK supports UWP app development for Windows 10 using C# and XAML.
• For UWP apps, the SDK supports only raw color and depth streaming and the blob tracking algorithm. Other UWP-specific algorithms are in development.
• UWP apps must statically include the SDK runtime files, so the SDK version is fixed at the time of development.
• The Session interface is not explicitly exposed in C# UWP.
• The camera coordinate system is slightly different for UWP apps; refer to the SDK manual for details.
• You cannot change the coordinate system in your UWP application.
• To map the coordinates between color and depth streams in UWP apps, use the Windows.Devices.Perception.PerceptionDepthCorrelatedCoordinateMapper interface.
• In the SDK manual, the (UWP) marking indicates UWP-specific interfaces or methods. The (+UWP) markings emphasize that the UWP interface is part of a function along with the other language interfaces.

Windows Desktop App Samples

Once the DCM and SDK are installed, reboot your computer and then ensure the camera is operating correctly by running one of the samples provided in the Intel RealSense SDK Sample Browser.

Click the Windows 10 Start button in the lower-left corner of the screen.

Select All Apps and then scroll to the Intel RealSense SDK folder (Figure 1).

Figure 1. Windows Start menu

Locate and run the Intel RealSense SDK Sample Browser. At the top of the SDK Sample Browser window (Figure 2) you’ll find a tab labelled “SR300 Samples” containing all of the Windows Desktop sample apps (i.e., code samples that run in Windows Desktop mode, not as UWP apps). You should familiarize yourself with these samples to understand the full capabilities of the SR300 camera.

Figure 2. SDK Sample Browser

UWP Components

The UWP software components provided in the Intel RealSense SDK are located in C:\Program Files (x86)\Intel\RSSDK\UWP. (Note: this is the default installation path; your file path may be different depending on how the SDK was installed.) The components are located in the following folders under \UWP:

• \ExtensionSDKs – Contains the DLLs that are referenced in a UWP application.
• \Samples  – Contains the DF_BlobViewer_UWP_CS and DF_StreamViewer_UWP_CS code samples.

Creating a UWP Project from Scratch

If you are new to UWP app development, first familiarize yourself with the basics of creating a UWP project from scratch. An informative C#/XAML “Hello, world” tutorial is available on this Microsoft website.

This tutorial provides a good starting point for learning how to create a simple app that targets the UWP and Windows 10. After completing the “Hello, world” tutorial and running your new UWP app, it will look something like the screen shown in Figure 3.

Figure 3. Hello, world! UWP application

When running in Debug mode you may notice a little black frame counter in the upper-left corner of your app. If you don’t want the counter to show, locate the following code in App.xaml.cs:

#if DEBUG

        if (System.Diagnostics.Debugger.IsAttached)

        {

            this.DebugSettings.EnableFrameRateCounter = true;

        }

#endif

You can either set the property to false:

this.DebugSettings.EnableFrameRateCounter = false;

Or simply comment-out the line:

// this.DebugSettings.EnableFrameRateCounter = true;

Configure the Development Environment for the Intel®RealSense™ Camera

To enable your app for the Intel RealSense camera, do the following:

• Enable Webcam under Capabilities in the App Manifest.
• Add references to the Intel RealSense SDK libraries.

In Solution Explorer, double-click Package.appxmanifest, and then click the Capabilities tab.

Locate and select the checkbox for Webcam, as shown in Figure 4.

Figure 4. Package.appxmanifest

Next you’ll need to reference the Intel RealSense SDK libraries:

• Intel.RealSense– The library containing the SDK essential instance implementation, such as algorithm management and streaming data from the cameras.
• Intel.RealSense.Blob– The library containing the SDK Blob Tracking module implementation.

Right-click on References in Solution Explorer, and then select Add Reference to open the Reference Manager.

Click the Browse button and navigate to the folders containing Intel.RealSense.winmd and Intel.RealSense.Blob.winmd. (These metadata files are located under C:\Program Files (x86)\Intel\RSSDK\UWP\ExtensionSDKs\.)

Click the Add button. The libraries appear under References in Solution Explorer.

Explore the UWP Samples

To learn more about how to integrate the Intel RealSense SDK capabilities in your app, open and build the two sample projects provided in the SDK:

• DF_BlobViewer_UWP_CS
• DF_StreamViewer_UWP_CS

Note: These samples are not available in the SDK Sample Browser app discussed earlier. They are located under C:\Program Files (x86)\Intel\RSSDK\UWP\Samples and should be copied to any writable directory in order to build them with Visual Studio 2015.

Summary

This articles presents a brief overview of developing UWP apps that integrate the Intel RealSense SDK. Stay tuned for more information as the SR300 camera becomes available.

About the Author

Bryan Brown is a software applications engineer in the Developer Relations Division at Intel. 

Can I redistribute libraries included with Intel® Parallel Studio XE 2016 with my application?

Yes. When you agree to the EULA for Intel® Parallel Studio XE you receive rights to redistribute portions of Intel® MKL, IPP, TBB and DAAL libraries with your application. However, the evaluation versions of Intel® Software Development products do not include redistribution rights.

Where do I find the redistributable packages for Intel® Parallel Studio XE 2016 for C++ and Fortran?  

The following articles contain links to the redistributable installation packages:

• Windows: https://software.intel.com/en-us/articles/redistributables-for-intel-parallel-studio-xe-2016-composer-edition-for-windows
• Linux: https://software.intel.com/en-us/articles/redistributables-for-intel-parallel-studio-xe-2016-composer-edition-for-linux
• OS X: https://software.intel.com/en-us/articles/redistributables-for-intel-parallel-studio-xe-2016-composer-edition-for-os-x
• All versions: https://software.intel.com/en-us/articles/intelr-composer-redistributable-libraries-by-version

Redistributable libraries for Intel® TBB, Intel® IPP and Intel® MKL are installed along with Intel® Parallel Studio XE.

What are the licensing terms of redistributing the libraries?

Subject to the terms of the EULA, you may redistribute an unlimited number of copies of the Redistributable files that are listed in the text files defined in the Redistributables section of the EULA.

Have Questions?

Please consult the Intel User Forums:

• Intel® C++ Compiler Forum
• Intel® Visual Fortran Compiler for Windows Forum

Intel has a multitude of awesome software development tools, including ones for media and graphics optimization. But sometimes, it's hard to figure out just which tool is the best one to use for your particular needs and usages.

Below you'll find a few insights to help you get to the right media tool faster, so you can focus on the really fun stuff - like optimizing your media solutions, applications, or video streaming. 

Abstract

Looking for tips on obtaining the best depth data from your 3D camera? This article will describe a few key characteristics of the Intel^® RealSense™ camera R200 and the Intel^® RealSense™ SDK that can affect depth data. You’ll also find a methodology for collecting raw data, including sample code (available from GitHub.com) as well as an attached zip file containing typical data.
We suggest testing under your specific lighting, texture, and distance parameters to provide some working boundaries for your scenes.

Overview

Figure 1 Intel(R) RealSense(TM) Camera R200 Reference Developer Kit

This analysis uses data obtained from the world facing Intel RealSense camera R200. The R200 is the first generation of Intel’s rear-facing 3D cameras based on active stereoscopic depth technology.  The camera implements an IR laser projector, a pair of stereoscopic IR cameras and a 2 MP RGB color camera.  You can purchase a developer kit for this camera from http://click.intel.com/intel-realsense-developer-kit-r200.html or a system with an integrated R200 camera.

Considerations - Camera and Object

The construction and behavior of the camera components can affect the resulting data, even in the case of fixed indoor lighting with stationary camera and target. Features and functions of the camera that affect depth data include:

• Temporal (frame-to-frame) and Spatial (within a frame) variation
• Sensor to sensor alignment and each sensor’s field of view (FOV)
• The environment’s temperature (effect on sensors)
• The environment’s ambient IR light
• Any other physical effect on a sensor including physical impact or vibration

The location and type of target, especially the object’s IR reflectivity and texture (sheen, dimples, and curvature) will also affect the depth data. For simplicity, this data collection used a fixed target of single texture.

These tests were performed using Windows* 8.1 64 bit with August update and v6 (R4) of the Intel RealSense SDK available at https://software.intel.com/en-us/intel-realsense-sdk/download.

Considerations - Geometry

For the simplest test, the camera was pointed directly at a flat target, a typical inside white wall with standard office fluorescent ceiling lighting.  The camera was mounted firmly, at a fixed height, on a rail that can travel linearly (one axis only) from 600-2100 mm from the target.  The distance is measured by a consumer-grade laser meter mounted flush with the front of the camera. 
The camera face is aligned parallel to the wall and the rail axis is aligned perpendicular to the wall.

Figure 2:  Test measurement set up with Intel® RealSense™ camera mounted for stabilization

While the camera moves between measurements, it was sitting still while collecting data.

In these tests, the motion of the rail induced significant vibrations that would affect the data so a large delay was introduced to allow the vibrations to cease before any frames were captured.  The rail motion, time delays, and data capture were automated to provide consistent, repeatable results (and to cut down on labor).

Tip: Alignment of the rail and camera with the target is critical. You can align the rail axis using physical measurement from the end of the rail to reference points on the wall. Aligning the camera face is more difficult. For these tests, we adjusted the camera alignment based on a spatially uniform pattern observed in the raw depth image from an Intel® RealSense™ SDK sample, with the contrast adjusted by scaling to make it easier for the human eye to see variations across the field of view.  See Figure 3 for a depiction of a non-spatially uniform pattern and Figure 4 for an aligned target image.  Note that the black area on the left is an artifact of stereoscopic imaging and does not reflect an alignment issue.

TIP:  All stereoscopic cameras will have areas where data is missing, due to the difference in field of view between the sensors. Figure 4 above shows such an area for the R200 camera in the black bar on the left.  The effect of this camera characteristic decreases with distance but needs to be considered at closer ranges.

Test Range

Using the above test configuration, we collected a series of raw depth buffers at distances ranging from 600 mm to 2100 mm.  600mm is the minimum distance for the Intel R200 camera and 2-3 meters is the maximum distance indoors (longer range outdoors).  Remember, range ultimately depends on multiple factors including lighting and object textures.

TIP:  Keep within the range of the depth camera. At distances less than 600 mm, the R200 camera may not return any depth data, depending on the configured resolution.

TIP:  For testing, keep the input scene simple (e.g. a flat wall), as it is easier to detect general trends. 

Quality and Precision Related to Distance

Our tests looked at two factors - the quality of the depth data and the precision of the depth data as functions of distance from the target.  For all depth cameras, accuracy and precision of depth measurements decreases as distance increases.

Tips:
For more accurate, precise data, stay as close to the object as possible. If moderate accuracy is needed, just stay within the recommended range of the camera.

Capturing Depth Data

A sample script is provided at https://github.com/IntelRealSense/rsdatacollector But to illustrate the flow, here’s a quick walkthrough of the basic steps to capture the camera data as individual frames using the Intel RealSense SDK:

1. Create and initialize an instance of PXCSenseManager (see PXCSenseManager::CreateInstance()).
2. Use sense manager to enable the depth and/or color stream  specifying resolution and frame rate.
          senseManager->EnableStream(PXCCapture::STREAM_TYPE_COLOR, 320, 240, 60);
          senseManager->EnableStream(PXCCapture::STREAM_TYPE_DEPTH, 320, 240, 30);
3. Use sense manager to acquire / lock an available frame.
           camStatus = senseManager->AcquireFrame(true);
4. Use sense manager to query the available samples
           PXCCapture::Sample *sample = senseManager->QuerySample();
5. Access the Sample's specific PXCImage members to extract the desired data .
               colorImage = sample->color;
               depthImage = sample->depth;
6. Use member functions of the PXCImage object, such as QueryInfo(), to get the parameters of the color or depth data.

NOTE: Understanding the image format, dimensions, and other details is critical when traversing the raw image buffers. The parameters are documented in the SDK and the sample code associated with this paper shows an example of how those parameters are used.

The data buffer is organized in planes and pixels and specific pitches that correspond to the parameters in the ImageInfo mentioned above.Once you understand the format of the data buffer, you can directly manipulate the values for various analysis or write it out to disk for analysis in other tools.

• 7. To access the raw image buffers, you must use the PCXImage::AcquireAccess() function passing it your PCXImage::ImageData pointer to store the buffer location. The planes and pitches of the ImageData should be initialized to empty arrays.
  
  8. After obtaining the desired image data, you must release access to the data buffer (PXCImage::ReleaseAccess()) and release the frame (PXCSenseManager::ReleaseFrame())

The Intel® RealSense™ SDK offers several abstractions for dealing with the image data, allowing easy access to and direct handling of the data buffers. These include the PXCPhoto::SaveXDM() function, which will simply write the depth enhanced JPEG file format. Additionally Background Segmentation (BGS), Enhanced Photography and Video (EPV) and Scene Perception (SP) modules in the SDK have options to provide additional filtering of the depth data. EPV and BGS share the same flags while SP flags are fixed.

Much more information is available in the SDK documentation at https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_devguide_introduction.html

Conclusion

Intel® RealSense™ Technology provides developers with depth data useful for extracting distance and size of objects from images and the ability to reconstruct a scene in 3D.  Since the quality of depth images varies due to many factors including time, lighting and distance, it is important to characterize the factors that may impact your application as described in this paper. Read the Intel RealSense SDK documentation to familiarize yourself with available SDK algorithms and sample code for capture, filtering, and processing.

Further Information

The Intel Developer Zone (IDZ) website for Intel RealSense technology includes the SDK and full documentation plus many articles, code samples, videos, blogs by software developers, links to events, and a forum for discussion and questions. Start at https://software.intel.com/realsense to check out the R200 camera features and videos.

About the Authors

	Tom Propst is a Software Engineer at Intel focusing on enabling new use cases in business environments. He received his B.S. in Electrical Engineering from Colorado State University. Outside of work, Tom enjoys playing bicycle polo and tinkering with electronics.
	Teri Morrison is a Software Architect at Intel focusing on the Intel® RealSense™ SDK. She has worked on NASA projects, graphics driver products and physics, graphics and perception computing samples. She received her M.S. in Applied Mathematics with minor in Computer Science from the University of Colorado.

Media for Mobile Getting Started Guide for Windows* platforms

Introduction

Media for Mobile, a feature of Intel® Integrated Native Developer Experience (Intel INDE) is a set of easy to use components and API for a wide range of media scenarios. It contains several complete pipelines for most popular use cases and enables you to add your own components to those pipelines.

The samples demonstrate how to incorporate the Media for Mobile into various applications for Windows* platforms.

Download the Media for Mobile samples from:

Media for Mobile Samples on GitHub

Media for Mobile Samples

System Requirements and List of available samples:

Host systems to develop applications for Windows*:

• Operating Systems: Microsoft* Windows* 8.1
• Intel® Integrated Native Developer Experience (Intel® INDE) 2015 for Windows* – Media For Mobile component
• IDE: Microsoft* Visual Studio* 2013 Update 2 or later

Windows* target systems:

• Hardware: IA-32 or Intel® 64 architecture processors
• Operating System: Microsoft* Windows* 8.1

Samples for Windows* RT* targets

• Transcode video
• Join Video
• Cut Video
• Video Effect
• Camera Capturing

The following tutorial guides you in your first steps with pre-built samples on...

Building and running samples for Windows* RT: steps needed to build the samples for Windows* RT in Microsoft* Visual Studio* 2013.

Media for Mobile is available for free with the Intel® INDE Starter Edition. Click here for more information.

Please see the list of open issues in Media for Mobile samples repository here.

Download WebAssembly article PDF  [PDF 263 KB]

Introduction

This first in a series of articles gives a high-level introduction to WebAssembly (wasm). While the language is not fully formed, with many changes to come, this article gives you a rough idea of the current state of wasm. We will provide follow-up articles to provide news of changes as they come along.

Wasm’s goal is to enable more performance for JavaScript*. It defines a new, portable, size- and load-time-efficient file format suitable for compilation to the web. It will use existing web APIs and is destined to become an integral part of Web technology. Although “WebAssembly” contains the word web in it, the language is not only destined uniquely for browsers, but the goal is to also provide the same technology to non-web usages. This opens the door to a lot more coverage and potential traction of wasm.

As with all dynamically typed languages, it is difficult to get anything close to native performance running JavaScript, so the ability to reuse existing native (C/C++) code in web pages was used for many years. Alternatives such as NaCl and PNaCl run in web pages alongside JavaScript. The most practical is asm.js, a JavaScript subset restricted to features that can be compiled into code approaching native performance.  None of these alternatives has been universally accepted in web browsers as the solution for performance and code reuse. Wasm is an attempt to fix that.

Wasm is not meant to replace JavaScript but instead to provide a means to attain near-native performance for key parts of an application, web-based or not. For this reason, browser, web, compiler, and general software engineers are working to create the definition of the technology. One of the goals is to define a new platform-independent binary code format for the web.

Near-native performing code on web pages could transform how new features are brought to the web. In native code, new features are often provided by an SDK, providing a native library for access. Web pages are not permitted to access those native libraries for security reasons. Web support for new capabilities often requires complex standardized APIs offered by the web browser to correct related issues; for example, JavaScript libraries are too slow.

With wasm, those standard APIs could be much simpler, and operate at a much lower level, with cached, multithreaded, and SIMD-capable dynamic link libraries of wasm providing functionality not currently possible. For example, instead of complex standard APIs for facial recognition or 3D image construction for use with a 3D camera, a much simpler standardized API could just provide access to the raw 3D data stream, with a wasm module doing the processing that a native SDK library now does. This would allow for downloading and caching of commonly used dynamic link libraries and rapid access to new capabilities from the web, long before the standardization process could complete.

This article provides an introduction to a very fast evolving technology. It will remain relatively high-level as many things are still evolving on the wasm specification.

More details about the high-level goals of wasm can be found here:

https://github.com/WebAssembly/design/blob/master/HighLevelGoals.md

General Overview

Design Versus Spec

The original design documents of wasm can be found in the following repository (repo), which contains these notable files:

• AstSemantics.md provides an overview of the format itself.
• MVP.md: what defines the Minimal Viable Product - requirements for the first iteration of wasm.
• HighLevelGoals.md: the high-level goals of wasm and the use cases it is trying to solve.

In order to precisely define, and also to verify and check the decisions recorded in the design document, the spec repo contains an OCaml interpreter of the wasm language. It also contains a test suite folder that has some initial wasm tests. The test suite ranges from general integer and floating-point calculations to memory operations.

A few other tools in the main wasm github repository use the same test suite to test for regression. For example, both wasm-to-llvm-prototype and wasmint use the same test suite for their regression tests.

Prototypes

The base github page for wasm contains various active projects. We mentioned a few of them in this article and there are many others of note. We recommend that the reader peruse the various projects to see where the wasm community is putting their efforts, but you can divide the repositories in five major groups:

• Design and specification: the “bible” of the wasm definition.
• binaryen: C/C++ to wasm and sometimes more.
• sexpr-wasm-prototype: wasm to a binary format.
• wasm-to-llvm-prototype, wasm-jit-prototype, wasmint: wasm to a runtime/executable code.
• polyfill-prototype-2: JavaScript to wasm.

Many of these repositories are proof-of-concept type repositories, meaning that they try to get something up and running to gather experience and don't necessarily represent the final result. The engineers working on the repositories often are playing around with wasm to determine how everything works. Examples are the various binary formats that are being tested such as polyfill-prototype-2 or the v8 binary format.

WebAssembly: An Initial View

Modules, the Larger Construction Blocks

Wasm defines modules as its highest construct, containing functions and memory allocation requests. A wasm module is a unit of distributable, executable code. Each module has its own separate linear memory space, imports and exports, and code. This could be an executable, a dynamic link library (in future versions of wasm) or code to execute on a web page (used where ECMAScript 6* modules can be used).

Though the current test files in the test suite allow multiple modules defined in the same file, it is currently thought that this will not be the case in the final version. Instead, it is probably going to be common to have a single big module for an entire program. Most C/C++ programs will therefore be translated into a single wasm module.

Functions

Wasm is statically typed with the return, and all parameters are typed. For example, this line from the i32.wast file in the test suite repository shows an addition of two parameters:

(func $add (param $x i32) (param $y i32) (result i32) (i32.add (get_local $x)
    (get_local $y)))

All are 32-bit integers. The line reads like this:

1. Declaration of a function named $add.
2. It has two parameters $x and $y, both are 32-bit integers.
3. The result is a 32-bit integer.
4. The function’s body is a 32-bit addition.
   • The left side is the value found in the local variable/parameter $x.
   • The right side is the value found in the local variable/parameter $y.
5. Since there is no explicit return node, the return is the last instruction of the function, hence this addition.

There will be more information about functions and wasm code later in this article.

Similar to an AST

Wasm is generally defined as an Abstract Syntax Tree (AST), but it also has some control-flow concepts and a definition of local variables allowing the handling of temporary calculations. The text format S-Expression, short for Symbolic Expression, is the current text format used for wasm (although it has been decided that it will not be the final wasm text representation).

However, it works well for explaining wasm in this document. If you ignore the general control flow constructs such as ifs, loops, and blocks, the wasm calculations are in AST format. However, for the following calculation:

(3 * x + x) * (3 * x + x)

You could see the following wasm code:

(i32.mul
       (i32.add (i32.mul (i32.const 3) (get_local 0)) (get_local 0))
       (i32.add (i32.mul (i32.const 3) (get_local 0)) (get_local 0))
 )

This means that the wasm-to-native compiler would have to do the common subexpression elimination to ensure good performance. To alleviate this, wasm allows the code to use local variables to store temporary results.

Our example can then become:

(set_local 1
       (i32.add (i32.mul (i32.const 3) (get_local 0)) (get_local 0))
)
 (i32.mul
       (get_local 1)
       (get_local 1)
 )

There are discussions about where the optimizations should lie:

• Between the original code, for example C/C++, to wasm
• Between wasm and the binary code used for the target architecture in the browser or in the non-web case

Memory

The memory subsystem for wasm is called a linear memory where the module can request a given size of memory starting at address 0. Loading and storing from memory can either use constants for simple examples or code but also variables containing addresses.

For example, this would store the integer 42 at the address location 0:

(i32.store (i32.const 0) (i32.const 42))

Wasm does define sign or zero extension for the memory operations. The operations can also define the alignment of the memory operation in case the architecture could take advantage of it for better code generation. Finally, the operation also has the offset parameter to permit loads of (for example) a structure field.

Wasm-to-LLVM Prototype

The wasm-to-LLVM prototype, the tool that I am contributing to wasm, provides a means to compile wasm code directly into x86 code via the LLVM compiler. Though wasm is intended for use in the web browser, there are plans to use wasm in non-web-based scenarios, as defined in the high-level goals of the language.

The general structure of the wasm-to-LLVM prototype is to parse the wasm test file using the open source tools flex and bison to construct an Intermediate Representation (IR). In this intermediate representation, there is a pass, and there most likely will be more in the future, that walks the IR before performing the code generation via the LLVM compiler.

Figure 1:wasm-to-LLVM prototype.

Figure 1 shows the base structure of the tool: it starts with using as input the temporary textual file format for wasm. This format is known as the S-expression. The S-expression is parsed by a lexical and semantic parser, implemented using the flex and bison tools. Once the S-expression is parsed, an internal Intermediate Representation (IR) is created and a pass system is used to change it slightly before generating the LLVM IR. Once in LLVM IR, the code is then sent to the LLVM optimizer before generating Intel® x86 code.

A First Wasm Example

Here we include one basic example: a simple sum of the values in an array. The example demonstrates how wasm is easy to understand and a few things to keep in mind.

Wasm is intended to be generated by a compiler, and C/C++ is a source language that will be first used to generate wasm. There is not a specific compiler that will transform C/C++ into wasm, although LLVM has already seen some wasm-related work committed to the project. We will likely see other compilers such as GCC and MSVC support the wasm language and environment. While writing wasm by hand will be rare, it is interesting to look at and understand how the language is meant to interact with the browser/OS and the underlying architecture.

Sum of an array

;; Initialization of the two local variables is done as before:
;;   local 1 is the sum variable initialized to 0
;;   local 2 is the induction variable and is set to the
;;   max element and is decremented per iteration
 (loop
        (if_else
          (i32.eq (get_local 2) (i32.const 0))
           (br 1)
            (block
              (set_local 1 (i32.add (get_local 1) (i32.load (get_local 2))))
              (set_local 2 (i32.sub (get_local 2) (i32.const 4)))
            )
        )
        (br 0)
      )

Note: this is not necessarily the optimal way of doing things. It is a useful example to demonstrate key parts of the wasm language and logic. A future article will explore various constructs that can better mimic the underlying hardware’s instructions.

As the example above shows, a loop is defined by a loop node. Loop nodes can define the start and exit block’s names or be anonymous as it is here. To better understand the code, the loop uses two local variables: Local 1 is the sum variable while walking the array, and local 2 is the induction variable being updated. Local 2 actually represents the pointer to the current cell to be added.

Here is the C-counterpart of the code:

// Initialization is done before
//   local_1 is the sum variable initialized to 0
//   local_2 is the induction variable and is set to
//   the max element and is decremented per iteration
do {
   if (local_2 == start) {
     break;
   }
   local_1 = local_1 + *local_2;
   local_2--;
} while(1);

The loops in wasm actually work like do-while constructs in C. They also are not implicitly looping back; we need to define an explicit branch at the end of the wasm loop. Further, the “br 0” node at the end says to branch to top of the loop; the 0 represents the level of the loop nest we want to go to from here.

The loop starts with checking if we want to do an extra iteration. If the test is true, we will do a “br 1”, which you might infer is to go out one level of the loop nest. In this case, since there is only one level, we leave the loop.

In the loop, notice that the code actually is using a decrementing pointer to reach the start of the array. In the C-version there is a convenient variable called start, representing the array the code is summing.

In wasm, since the memory layout starts at address 0 and since this is the only array of this overly simplified example, it is arbitrarily determined that the start address of the array would be 0. If we put the array anywhere else, this comparison would look more like the C-version and compares the local variable 2 with a parameter-passed offset.

Notice the difference in handling the induction variable’s update between the C and the wasm versions. In C, the language allows the programmer to simply update the pointer by one and it will transform this into an actual decrement into four later down the line. In wasm, we are already at a very low level, hence the decrement of four is already there.

Finally, as we showed that (get_local 2) is the loop counter, (get_local 1) is the actual sum of the vector. Since we are doing a sum in 32-bit, the operation is using the i32.add and i32.load opcodes. In this example, the vector we are summing is at the beginning of the linear memory region.

Wasm-to-llvm Code Generation

The wasm-to-llvm-prototype generates the following loop code quite easily:

.LBB4_2:
  movl  %edi, %edx
  addl  (%rdx,%rcx), %eax
  addl  $-4, %edi
  jne .LBB4_2

This is quite a tight loop when you consider the original wasm text format code. If you consider the following equivalent C-version:

for (i = 0; i < n; i++) {
    sum += tab[i];
}

The GNU* Compiler Collection (GCC), version 4.7.3, produces in the -O2 optimization level, the following code is similar in its logic:

.L11:
  addl  (%rdi,%rdx,4), %eax
  addq  $1, %rdx
  cmpl  %edx, %esi
  jg  .L11

In the wasm case, we have generated a countdown loop and use the subtraction result directly as the condition for the loop jump. In the GCC case, a comparison instruction is required. However, the GCC version does not use an additional move instruction that the wasm with LLVM uses.

In O3, GCC vectorized the loop. What this shows is that the wasm-to-llvm prototype still has a bit of work in order to generate the optimal code (vectorized or not). This will be described in a future article.

Conclusion

This article introduced you to the wasm language and showed a single simple example of a sum of an array. Future articles will dive more into detail about wasm and its different existing tools.

Wasm is a new language and a lot of tools are being developed to help determine its potential, to support it, and to understand what it can and cannot do. There are many elements about wasm that need to be determined, explained, and explored.

About the Author

Jean Christophe Beyler is a software engineer in the Intel Software and Solutions Group (SSG), Systems Technologies & Optimizations (STO), Client Software Optimization (CSO). He focuses on the Android compiler and eco-system but also delves into other performance related and compiler technologies.

This sample uses a native Unity plug-in to increase performance of displaying Intel® RealSense™ SDK image data by bypassing the C# layers of the SDK. Image data is uploaded to the GPU directly through the graphics API.

The sample is available at https://github.com/GameTechDev/UnityRealSenseNativeTexture.

The sample supports Direct3D* 9, Direct3D* 11, and OpenGL*, but it could be modified to support other APIs. This plug-in is based on the native plug-in example provided by Unity. That sample, and more information on native Unity plug-ins, can be found here.

Figure 1. Screenshot from the sample running in Unity*.

Sample Contents

This sample includes:

• The plug-in source code
• A sample Unity project that uses the plug-in

How It Works

The process of this native texture copy in Unity is as follows:

1. The application obtains a PXCMImage object from the Intel® RealSense™ SDK and uses the QueryNativePointer function to obtain the native PXCImage pointer.
2. The PXCImage pointer and the destination Unity Texture2D pointer are passed to the native plug-in.
3. When the native plug-in’s callback is invoked on Unity’s render thread, the data from the PXCImage image is copied to the native texture using the appropriate graphics API.

Requirements

• Intel® RealSense™ SDK R5 or higher is recommended. Previous SDKs can be used but the developer will need to modify the PXCCLR project to add support for the QueryNativePointer function.
• Unity 5.2 or higher is recommended. The native plug-in was not tested on previous versions.
• Visual Studio* 2013 or higher. Previous versions can be used, but haven’t been configured.

Running the Sample

The sample includes a Unity project called SampleProject which demonstrates using the plug-in to display the color stream of an Intel® RealSense™ camera.

Follow these steps to run the sample:

1. Download the Intel® RealSense™ SDK version – You will need Intel RealSense SDK version R5 or greater.
2. Update the SampleProject’s SDK DLLs - Run the UpdateSDKDlls.bat in the SampleProject folder to copy the required managed DLLs from the SDK into the SampleProject.
3. Build the native plug-in – Open and build NativeRSTextureCopy.sln which can be found in within the src folder. A post-build step will copy the plug-in into the SampleProject folder.
4. Open and Run the SampleProject in Unity.

Integrating the Plug-in

Follow these steps to integrate the native plug-in into an existing Unity application that uses Intel® RealSense™ SDK:

1. Build the native plug-in – The plug-in must be rebuilt for the SDK the application is using. Open the Visual Studio solution and build the Release configuration for the desired target architecture.
2. Add UseTexturePlugin.cs to the project – Copy the UseTexturePlugin.cs script from the SampleProject folder into your project, and attach it to a persistent game object in the scene. This script is responsible for pumping the native render callback event as well as providing an interface to the native plug-in functionality.
3. Pass PXCMImages to the plug-in – Replace calls in your application to PXCMImage.ToTexture2D with calls to UseTexturePlugin.CopyPXCImageToTexture.
4. Handle API specific formatting – Depending on the graphics API, the texture may be displayed flipped vertically, or with the colors inverted. The NativeRGBImage.shader file demonstrates how to correct this formatting issue.
5. Notify on shutdown - Call the UseTexturePlugin.RealSenseShutdown function before shutting down the Intel® RealSense™ SDK Sense Manager.

Important Notes

• Because this is a native plug-in linked against a specific version of the Intel® RealSense™ SDK, you must recompile the plug-in when switching to different SDK versions. Also, be sure to compile it for all CPU architectures you plan on supporting.
• The QueryNativePointer function was added to the C# API in the R5 release to allow access to managed SDK objects’ underlying native objects. This sample will still work on previous SDKs, but you will need to add the QueryNativePointer functionality to the PXCCLR project in the installed SDK.