06.Programming Media on Windows Phone

Programming Media on WP
Nguyen Tuan | Microsoft Certified Trainer

Agenda
• The Windows Phone Camera
• Taking still images
• Manipulating the video stream
• The Windows Phone Microphone
• The Windows Phone Sensors
• The Motion sensor
• Video Content
• Text to Speech and Speech to Text in windows Phone 8

The Camera
• There are a number of ways the camera can be used by an application
• The application can launch the CameraCaptureTask chooser to allow the user to take
a photograph
• The application can use the PhotoCamera class to capture photos or stream video
data from the camera
• Can use this for product recognition or augmented reality
• Use the PhotoCaptureDevice class for advanced photo capture and
AudioVideoCaptureDevice for advanced video capture
• A real-time video processing application can be registered as a “Lens”
• It can be selected by the user from a menu of available lens types
• The Lens application will provides a viewfinder display and performs video processing on the signal from the
camera

cameraTask = new CameraCaptureTask();
cameraTask.Completed += new EventHandler<PhotoResult>
(cameraTask_Completed);
cameraTask.Show();
Capturing a photo
• This task launches the camera so that the user can take a picture
• The cameraTask_Completed event fires when the picture is taken
• The emulator will return an image containing a small block

void cameraCapture_Completed(object sender,
PhotoResult e)
{
if (e.TaskResult == TaskResult.OK)
{
photoImage.Source = new BitmapImage(
new Uri(e.OriginalFileName));
}
}
Capture complete
• When capture has been made, completed method is executed
• This version just displays the image on the screen
• You can access the photo stream data to save it

In application capture
• It is also possible to capture an image from within your application
• An application can also display a viewfinder
• The application can also access the video data directly and use this for
augmented reality or to create a customised viewfinder
• eg. Barcode scanners
• The PhotoCamera class provides camera control and access to the
camera video screen
7

The PhotoCamera class
• This creates a camera and binds a handler to the captured event
• The viewfinderBrush source is set to the camera
using Microsoft.Devices;
...
PhotoCamera camera;
...
camera = new PhotoCamera();
//Set the VideoBrush source to the camera
viewfinderBrush.SetSource(camera);
camera.CaptureImageAvailable +=
new EventHandler<ContentReadyEventArgs>
(camera_CaptureImageAvailable);

Displaying the Viewfinder
• This is the rectangle in the xaml for the camera viewfinder page
• This will display the viewfinder on the screen
• The source for the viewfinder brush is set to the camera
<Rectangle Width="320" Height="240" HorizontalAlignment="Left" >
<Rectangle.Fill>
<VideoBrush x:Name="viewfinderBrush" />
</Rectangle.Fill>
</Rectangle>

Initiating the Capture
• This is the event handler for the photo button
• It asks the camera to take a picture
• The current camera settings are used for this
• You can override these (for example turn the flash on or off) by setting
properties on the camera instance
private void PhotoButton_Click(object sender, RoutedEventArgs e)
{
camera.CaptureImage();
}

Saving the image
• This saves the image in the camera roll
using Microsoft.Xna.Framework.Media;
...
void camera_CaptureImageAvailable(object sender,
ContentReadyEventArgs e)
{
Deployment.Current.Dispatcher.BeginInvoke(delegate()
{
string fileName = DateTime.Now.Ticks.ToString()
+ ".jpg";
MediaLibrary library = new MediaLibrary();
library.SavePictureToCameraRoll(fileName e.ImageStream);
});
}

Displaying the image
• This saves the image into a BitmapImage which is displayed on the screen in an
<Image>
using System.Windows.Media.Imaging;
...
void camera_CaptureImageAvailable(object sender,
ContentReadyEventArgs e)
{
{
BitmapImage b = new BitmapImage();
b.CreateOptions = BitmapCreateOptions.None;
b.SetSource(e.ImageStream);
PictureImage.Source = b;
});
}

Saving to the Local Folder
• This saves the image in the local folder
using System.IO;
using System.IO.IsolatedStorage;
...
using (IsolatedStorageFile isStore =
IsolatedStorageFile.GetUserStoreForApplication())
{
using (IsolatedStorageFileStream targetStream =
isStore.OpenFile(fileName, FileMode.Create, FileAccess.Write))
{
WriteableBitmap bitmap = new WriteableBitmap(b);
bitmap.SaveJpeg(targetStream, bitmap.PixelWidth, bitmap.PixelHeight, 0, 100);
}
}

Creating a Lens
• A Lens is a custom camera application
which can be accessed from within the
camera application
• An application is flagged as a Lens
application by setting a flag in the
manifest and providing icons that can be
used to browse for the Lens when the
camera is in use
• I’ve created a FunkyCamera lens
application which I have registered in this
way
14

Creating a Lens application
• This text must be added to the WMAppManifest.xml file for the
application, just after the <Tokens> section
• There is no GUI for this alteration, you have to edit the XML directly
<Extensions>
<Extension ExtensionName="Camera_Capture_App"
ConsumerID="{5B04B775-356B-4AA0-AAF8-6491FFEA5631}"
TaskID="_default" />
</Extensions>

Adding the Lens Icons
• Three Icons are required, one for each Windows Phone screen size
• WVGA 173 × 173 AssetsLens.Screen-WVGA.png
• 720p 259 × 259 AssetsLens.Screen-720p.png
• WXGA 277 × 277 AssetsLens.Screen-WXGA.png
• They are all placed in the Assets folder of the application
• Use a transparent background to match the Windows Phone color scheme
8/16/2014 16

Lens Startup
• You can create a URI Mapper to direct the application to the page that
implements the viewfinder for the Lens
• This can use the string “Viewfinderlaunch” in the destination uri that is
activated when the user selects the lens application
• Alternatively, if the program only contains a single page this page is
displayed
• If the user “backs out” of the lens application they will be returned to
the camera
8/16/2014 17

Image Processing in a Lens
• The PhotoCamera class is used to provide access to the video stream
• This can be used by applications to perform image processing on the live
data
• Augmented reality
• Scanning applications
• We are going to use it to make the funky camera display a funky image
18

Funky Image Processing
• This separates out the primaries and adds an offset to each
• It is called for each pixel in the image
internal int FunkyColor(int color)
{
int a = color >> 24;
int r = (color & 0x00ff0000) >> 16;
int g = (color & 0x0000ff00) >> 8;
int b = (color & 0x000000ff);
r += redOffset;
g += greenOffset;
b += blueOffset;
return ((a & 0xFF) << 24) | ((r & 0xFF) << 16) |
((g & 0xFF) << 8) | (b & 0xFF);
}

Starting the Camera
• This creates the camera and the bitmap that will contain the image
processed output
• It binds to the event fired when the camera is ready
camera = new Microsoft.Devices.PhotoCamera();
// Create the destination for the processed image
wb = new WriteableBitmap(640, 480);
this.ProcessedImage.Source = wb;
// Start the image pump when the camera is ready
camera.Initialized +=
new EventHandler<CameraOperationCompletedEventArgs>
(camera_Initialized);

Starting the Camera
• When the camera is ready we start the thread that will pump frames into
our image processor
• This will run alongside our application
void camera_Initialized(object sender, CameraOperationCompletedEventArgs e)
{
pumpARGBFrames = true;
ARGBFramesThread =
new System.Threading.Thread(PumpARGBFrames);
ARGBFramesThread.Start();
}

Getting the Image Data
• This code grabs the preview buffer from the camera and processes it
int[] ARGBPx = new int[640 * 480];
...
captureEvent.WaitOne();
pauseFramesEvent.WaitOne(); //Thread sync with camera
//Copies the current viewfinder frame into a buffer
camera.GetPreviewBufferArgb32(ARGBPx);
//Conversion to funky colours
for (int i = 0; i < ARGBPx.Length; i++)
{
ARGBPx[i] = FunkyColor(ARGBPx[i]);
}

Drawing the Image Data
• This code writes the processed pixels back to a writeable bitmap that is
displayed on the screen
private WriteableBitmap wb;
...
pauseFramesEvent.Reset();
{
//Copy to WriteableBitmap
ARGBPx.CopyTo(wb.Pixels, 0);
wb.Invalidate();
pauseFramesEvent.Set();
});

• You can also create an application with an associated “resource intensive”
background task that can auto-upload pictures that the user may take to a
cloud storage service
• This is a resource intensive background process and may run when the phone
is on charge
• The user can select your app from the Camera Settings
• The application must set the extension shown above and display a settings
page where the user can set authentication and upload options
Creating an Auto-Uploader for Photos
<Extensions>
<Extension ExtensionName="Photos_Auto_Upload"
ConsumerID="{5B04B775-356B-4AA0-AAF8-6491FFEA5632}"
TaskID="_default" />
</Extensions>

The Microphone
• The Windows Phone microphone can capture 16 bit audio
• Microphone input is managed as part of the XNA framework
• It is possible to record sound, process the audio and either store or
replay it
• There is a complete example of how to do this on MSDN

Sensors
8/16/2014 Microsoft confidential 28

Sensors Available
• There are a number of different sensors:
• Accelerometer
• Compass
• Gyroscope
• Inclinometer
• Orientation
• All the sensors are used in the same way:
• They will fire an event when they have a reading
• Two APIs for managed code developers:
• Microsoft.Devices.Sensors (Windows Phone OS 7.1 API set)
• Windows.Devices.Sensors (Windows Phone Runtime)

The Windows Runtime Sensors Library
• The Windows Phone Runtime sensors APIs are compatible with sensor usage in WinRT on
Windows 8
• Accelerometer – returns G-force values with respect to the x, y, and z axes
• All phones have an accelerometer
• Inclinometer - returns pitch, roll, and yaw values that correspond to rotation angles
around the x, y, and z axes, respectively
• The inclinometer readings are derived from multiple sensors
• Gyrometer - returns angular velocity values with respect to the x, y, and z axes
• Compass - returns a heading with respect to True North and, possibly, Magnetic North
• OrientationSensor - returns a rotation matrix and a Quaternion that can be used to
adjust the user's perspective in a game application
• Combines the data from the accelerometer, compass, and gyrometer – known as “Sensor
Fusion”
using Windows.Devices.Sensors;

Determining Sensor Availability
• All the sensor classes have a GetDefault() method
• This method only returns values for hardware that has been integrated into the
computer by the manufacturer
• Returns null if the sensor is not available on that device
• All phones will have an accelerometer
// Determine whether we have a gyro on the phone
_gyrometer = Gyrometer.GetDefault();
if (_gyrometer != null)
{
// Establish the report interval (units are milliseconds)
_gyrometer.ReportInterval = 100;
_gyrometer.ReadingChanged += _gyrometer_ReadingChanged;
}
else
{
MessageBox.Show("No gyrometer found");
}

Starting and Stopping a Sensor
• Application must set the report interval to a non-zero value prior to registering
an event handler or calling GetCurrentReading to activate it
• When finished with the sensor, set it to zero
• Check the MinimumReportInterval property
• Setting a value below the minimum supported interval will either trigger an exception or
have undefined results.
• Sensor driver will determine the actual report interval
// Establish the report interval (units are milliseconds)
uint reportInterval = 100;
if (_gyrometer.MinimumReportInterval > reportInterval)
{
reportInterval = _gyrometer.MinimumReportInterval;
}
_gyrometer.ReportInterval = reportInterval;

Using the Sensor ReadingChanged event
• Register the ReadingChanged event handler to obtain sensor readings
• Must set the ReportInterval property first
_gyrometer.ReportInterval = 100;
_gyrometer.ReadingChanged += _gyrometer_ReadingChanged;
…
private void _gyrometer_ReadingChanged(Gyrometer sender, GyrometerReadingChangedEventArgs args)
{
Dispatcher.BeginInvoke(() =>
{
GyrometerReading reading = args.Reading;
X_Reading.Text = String.Format("{0,5:0.00}", reading.AngularVelocityX);
Y_Reading.Text = String.Format("{0,5:0.00}", reading.AngularVelocityY);
Z_Reading.Text = String.Format("{0,5:0.00}", reading.AngularVelocityZ);
});
}

Getting Readings By Polling a Sensor
• An application can poll the sensor for the current reading as an alternative to
registering a ReadingChanged event handler
• The preferred alternative for an application that updates its user interface at a specific frame rate
• Must still establish a desired ReportInterval before polling in order to activate the
sensor
// Alternative to ReadingChanged event, call GetCurrentReading() to poll the sensor
GyrometerReading reading = _gyrometer.GetCurrentReading();
if (reading != null)
{
X_Reading.Text = String.Format("{0,5:0.00}", reading.AngularVelocityX);
Y_Reading.Text = String.Format("{0,5:0.00}", reading.AngularVelocityY);
Z_Reading.Text = String.Format("{0,5:0.00}", reading.AngularVelocityZ);
}

Video on the Phone
• An application can contain a single MediaElement that can play video
• The sample above plays a resource file that is part of the project
containing the application
• You can find a list of supported codecs here:
http://msdn.microsoft.com/en-us/library/ff462087.aspx
<MediaElement Name= "MediaPlayback"
Source= "myvideo.wmv" AutoPlay="True"/>

Streaming Video on the Phone
• If you want to stream from the internet, just replace the source file with a
url
• This implementation has the location hard coded into the XAML
• You can also do this under program control
<MediaElement Name= "MediaPlayback"
Source="http://mschannel9.vo.msecnd.net/o9/mix/09/wmv/key01.wmv"
AutoPlay="True"/>

Controlling Playback
• The MediaElement exposes methods that can be used to control the
media playback
• An application can also determine the properties of the media stream
• To determine if it can be paused for example
private void pauseButton_Click(object sender, RoutedEventArgs e)
{
MediaPlayback.Pause();
}

Smooth Streaming
• Windows Phone also supports Smooth Streaming
• This is an adaptive streaming that manages the quality of the video signal in
response to the abilities of the network connection
• Playback quality is managed in real time to handle changes in network
performance during viewing
• It uses a server side plugin and client code on the viewing device
• You can download the viewing software here:
http://smf.codeplex.com

Speech Support
• Windows Phone 7.x had voice support built into the operating system
• Programs and phone features could be started by voice commands e.g “Start
MyApp”
• Incoming SMS messages could be read to the user
• The user could compose and send SMS messages
• Windows 8 builds on this to allow applications to make use of speech
• Applications can speak messages using the Speech Synthesis feature
• Applications can be started and given commands
• Applications can accept commands using voice input
• Speech recognition requires an internet connection, but Speech Synthesis
does not
43

Enabling Speech Synthesis
• If an application wishes to use speech
output the
ID_CAP_SPEECH_RECOGNITION
capability must be enabled in
WMAppManifest.xml
• The application can also reference the
Synthesis namespace
8/16/2014 45
using Windows.Phone.Speech.Synthesis;

Simple Speech
• The SpeechSynthesizer class provides a simple way to produce speech
• The SpeakTextAsync method speaks the content of the string using the
default voice
• Note that the method is an asynchronous one, so the calling method
must use the async modifier
• Speech output does not require a network connection
8/16/2014 46
async void CheeseLiker()
{
SpeechSynthesizer synth = new SpeechSynthesizer();
await synth.SpeakTextAsync("I like cheese.");
}

Selecting a language
• The default speaking voice is selected automatically from the locale set
for the phone
• The InstalledVoices class provides a list of all the voices available on the
phone
• The above code selects a French voice
8/16/2014 47
// Query for a voice that speaks French.
var frenchVoices = from voice in InstalledVoices.All
where voice.Language == "fr-FR"
select voice;
// Set the voice as identified by the query.
synth.SetVoice(frenchVoices.ElementAt(0));

Speech Synthesis Markup Language
• You can use Speech Synthesis Markup Language (SSML) to control the spoken
output
• Change the voice, pitch, rate, volume, pronunciation and other characteristics
• Also allows the inclusion of audio files into the spoken output
• You can also use the Speech synthesizer to speak the contents of a file
8/16/2014 49
<?xml version="1.0" encoding="ISO-8859-1"?>
<speak version="1.0"
xmlns=http://www.w3.org/2001/10/synthesis xml:lang="en-US">
<p> Your <say-as interpret-as="ordinal">1st</say-as> request was for
<say-as interpret-as="cardinal">1</say-as> room on
<say-as interpret-as="date" format="mdy">10/19/2010</say-as> ,
arriving at <say-as interpret-as="time" format="hms12">12:35pm</say-as>.
</p>
</speak>

Controlling Applications using
Voice Commands
50

Using Voice command
• The Voice Command feature of Windows Phone 7 allowed users to start
applications
• In Windows Phone 8 the feature has been expanded to allow the user to
request data from the application in the start command
• The data will allow a particular application page to be selected when the
program starts and can also pass request information to that page
• To start using Voice Commands you must Create a Voice Command
Definition (VCD) file that defines all the spoken commands
• The application then calls a method to register the words and phrases
the first time it is run
8/16/2014 51

The Fortune Teller Program
• The Fortune Teller program will
tell your future
• You can ask it questions and it
will display replies
• It could also speak them
• Some of the spoken commands
activate different pages of the
application and others are
processed by the application
when it starts running
8/16/2014 52

<CommandPrefix> Fortune Teller </CommandPrefix>
<Example> Will I find money </Example>
<Command Name="showMoney">
<ListenFor> [Will I find] {futureMoney} </ListenFor>
<Feedback> Showing {futureMoney} </Feedback>
<Navigate Target="/money.xaml"/>
</Command>
<PhraseList Label="futureMoney">
<Item> money </Item>
<Item> riches </Item>
<Item> gold </Item>
</PhraseList>
The Voice Command…
• This is the “money” question:
“Fortune Teller Will I find money”
8/16/2014 53

</Command>
<Item> gold </Item>
</PhraseList>
The Voice Command…(cnt)
8/16/2014 54
•This is the phrase the
user says to trigger the
command
•All of the Fortune Teller
commands start with this
phrase

</Command>
<Item> gold </Item>
</PhraseList>
8/16/2014 55
•This is example text that
will be displayed by the
help for this app as an
example of the
commands the app
supports

</Command>
<Item> gold </Item>
</PhraseList>
8/16/2014 56
•This is the command
name
•This can be obtained
from the URL by the
application when it starts

</Command>
<Item> gold </Item>
</PhraseList>
8/16/2014 57
•This is the example for
this specific command

</Command>
<Item> gold </Item>
</PhraseList>
8/16/2014 58
•This is the trigger phrase
for this command
•It can be a sequence of
words
•The user must prefix this
sequence with the words
“Fortune Teller”

</Command>
<Item> gold </Item>
</PhraseList>
8/16/2014 59
•This is the phraselist for
the command
•The user can say any of
the words in the phraselist
to match this command
•The application can
determine the phrase
used
•The phraselist can be
changed by the application
dynamically

</Command>
<Item> gold </Item>
</PhraseList>
8/16/2014 60
•This is the spoken
feedback from the
command
•The feedback will insert
the phrase item used to
activate the command

</Command>
<Item> gold </Item>
</PhraseList>
8/16/2014 61
•This is the url for the page
to be activated by the
command
•Commands can go to
different pages, or all go to
MainPage.xaml if required

</Command>
<Item> gold </Item>
</PhraseList>
8/16/2014 62
•These are the phrases that
can be used at the end of
the command
•The application can modify
the phrase list of a
command dynamically
•It could give movie times
for films by name

Installing a Voice Command
• The VCD file can be loaded from the application or from any URI
• In this case it is just a file that has been added to the project and marked as
Content
• The VCD can also be changed by the application when it is running
• The voice commands for an application are loaded into the voice
command service when the application runs
• The application must run at least once to configure the voice commands
8/16/2014 63
async void setupVoiceCommands()
{
await VoiceCommandService.InstallCommandSetsFromFileAsync(
new Uri("ms-appx:///VCDCommands.xml", UriKind.RelativeOrAbsolute));
}

Launching Your App With a Voice Command
• If the user now presses and holds the Windows button, and says:
Fortune Teller, Will I find gold?
the Phone displays “Showing gold”
• It then launches your app and navigates to the page associated with this
command, which is /Money.xaml
• The query string passed to the page looks like this:
"/?voiceCommandName=showMoney&futureMoney=gold&reco=Fortune%20Teller%Will%20I%20find%20gol
d"
8/16/2014 64
Command Name Phaselist
Name
Recognized
phrase
Whole phrase as it
was recognized

Handling Voice Commands
• This code runs in the OnNavigatedTo method of a target page
• Can also check for the voice command phrase that was used
8/16/2014 65
if (e.NavigationMode == System.Windows.Navigation.NavigationMode.New) {
if (NavigationContext.QueryString.ContainsKey("voiceCommandName")) {
string command = NavigationContext.QueryString["voiceCommandName"];
switch command) {
case "tellJoke":
messageTextBlock.Text = "Insert really funny joke here";
break;
// Add cases for other commands.
default:
messageTextBlock.Text = "Sorry, what you said makes no sense.";
break;
}
}
}

Identifying phrases
• The navigation context can be queried to determine the phrase used to trigger
the navigation
• In this case the program is selecting between the phrase used in the “riches”
question
8/16/2014 66
<Item> gold </Item>
</PhraseList>
string moneyPhrase = NavigationContext.QueryString["futureMoney"];

Review
• There are two types of scheduled notifications, Alarm and Reminder
• An Alarm allows you to specify a sound file to play when the notification is launched
• When you create a Reminder, you can specify a deep link URI
• Applications can capture images and video feeds from the camera
• Applications can create “Lens” behaviours that are accessed from the in-phone camera
• Applications can provide upload behaviors to upload photographs
• Applications can use sensors and determine which of them are present
• The MediaContent element provides for video playback
68

06.Programming Media on Windows Phone

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (18)

Similaire à 06.Programming Media on Windows Phone

Similaire à 06.Programming Media on Windows Phone (20)

Plus de Nguyen Tuan

Plus de Nguyen Tuan (11)

06.Programming Media on Windows Phone