Using Azure Cognitive Services with NativeScript

Using
Azure
Cognitive
Services
with {N}
@Sherrrylst

Hello!
I’m Sherry List
Azure Developer Technical Lead, Microsoft
You can find me at @SherrryLst

Artificial Intelligence
AI is the simulation of human intelligence
processes by machines. These
processes include, reasoning,
remembering, learning and self-
correction.
@Sherrrylst

Machine Learning
@Sherrrylst
Machine learning is a field of computer
science that gives computers the ability
to learn without being explicitly
programmed.
Arthur Samuel, 1959

Machine learning techniques
@Sherrrylst
• Artificial Neural Networks
• Deep Learning
• Bayesian Networks
• Clustering

Artificial Intelligence (AI)
The bigger picture
Machine
Learning
Artificial Neural Networks
Deep Learning
Bayesian Networks
Clustering
@Sherrrylst

Machine
Learning 101
@Sherrrylst

Machine Learning 101
@Sherrrylst
Data

@Sherrrylst
Data
Patterns

@Sherrrylst
Data
Patterns
Algorithm
(Deep learning, clustering, …)

@Sherrrylst
Data
Patterns
Algorithm
Find patterns

@Sherrrylst
Data
Patterns
Model
Algorithm
Find patterns

@Sherrrylst
Data
Patterns
Model
Algorithm
Find patterns Recognizes patterns

@Sherrrylst
Data
Patterns
Model
Algorithm
Find patterns Recognizes patterns
Training

@Sherrrylst
Data
Patterns
Model
Algorithm
Find patterns Recognizes patterns Application
Training

@Sherrrylst
Data
Patterns
Model
Algorithm
Challenges
Data preparation

@Sherrrylst
Data
Patterns
Model
Algorithm
Challenges
Data preparation Create and test algorithms

@Sherrrylst
Algorithm
Find patterns
Challenges
Create and test algorithms
Chihuahua or muffin?

@Sherrrylst
Data
Patterns
Model
Algorithm
Challenges
Data preparation Create and test algorithms Expose the model to app

Is there a better
way?
@Sherrrylst

Azure
Cognitive
Services
@Sherrrylst

@Sherrrylst
Application
The easier way
Azure Cognitive
Services
REST API

Azure Cognitive Services === machine-learning-as-a-service
@Sherrrylst

Azure Cognitive Services
Vision Speech Language Knowledge Search
@Sherrrylst

Azure Cognitive Services
Vision Speech Language Knowledge Search
@Sherrrylst
Cognitive Services Labs

• Computer vision
@Sherrrylst
Cognitive Services - Vision

• Computer vision
• Video Indexer
@Sherrrylst

• Computer vision
• Video Indexer
• Face
@Sherrrylst

• Computer vision
• Video Indexer
• Face
• Content Moderator
@Sherrrylst

• Computer vision
• Video Indexer
• Face
• Custom Vision
@Sherrrylst

• Speech to Text
• Text to Speech
• Speaker recognition (Preview)
• Speech Translation
@Sherrrylst
Cognitive Services - Speech

• Language Understanding
• Bing Spell Check
• Translator Text
• Text Analytics
@Sherrrylst
Cognitive Services - Language

@Sherrrylst
• QnA Maker
Cognitive Services - Knowledge

Cognitive Services - Search
@Sherrrylst
• Bing Custom Search
• Bing Web Search
• Bing Video Search
• Bing Image Search
• Bing Local Business Search (Preview)
• Bing Visual Search
• Bing Entity Search
• Bing News Search
• Bing Auto Suggest

Cognitive Services Labs
@Sherrrylst
Labs provides developers with an early
look at emerging Cognitive Services
technologies.

How to use Cognitive Services?
@Sherrrylst
1. Create service in Azure Portal
(https://portal.azure.com/)

@Sherrrylst
2. Customize and train model in the
portal (Optional)

@Sherrrylst
portal (Optional)
3. Call the REST API from your app

@Sherrrylst
portal (Optional)
3. Call the REST API from your app
4. Parse the JSON response

Create a Cognitive Services account in the
Azure portal
@Sherrrylst

@Sherrrylst
Cognitive Services account - Face
Face API

@Sherrrylst
Cognitive Services account

@Sherrrylst
• URL : {Endpoint}/face/v1.0/detect
• URI Parameters
• Location
• returnFaceLandmarks (Boolean)
• returnFaceAttributes
• age,gender,smile,glasses,emotion,facialHair
• Headers
• Content-Type: application/octet-stream | application/json
• Ocp-Apim-Subscription-Key
Face API - Detect
Face API Documentation

@Sherrrylst
Face API - Detect
Face API Documentation

@Sherrrylst
Face API – Mood Analyzer
Source code
async runFaceDetect() {
this.dataSource = [];
const imageAsset = await takePicture(
{
width: 300,
height: 500,
keepAspectRatio: true,
saveToGallery: false
});
const filePath = await this.getFilePath(imageAsset);
const imageFile = File.fromPath(filePath);
this.pictureFromCamera = imageAsset;
this.result = await this.sendRequest(imageFile);
this.dataSource = getEmotionData(this.result);
}

@Sherrrylst
Face API – Posting data (Part 1)
Source code
sendRequest(file: File): Promise<any> {
return new Promise((resolve, reject) => {
const ses = session('image-upload');
const request = {
url: this.faceUrl,
method: 'POST',
headers: {
'Content-Type': 'application/octet-stream',
'Ocp-Apim-Subscription-Key': '7d30f6357121430e91b3f8439b8205c6',
},
description: 'Uploading ' + file.name
};
const task = ses.uploadFile(file.path, request);
...
});
}

@Sherrrylst
Face API – Posting data (Part 2)
Source code
sendRequest(file: File): Promise<any> {
return new Promise((resolve, reject) => {
...
task.on('error',
e => reject('error ' + e.responseCode + ' code.')
)
task.on('responded',
e => {
const data = JSON.parse(e.data);
return resolve(data);
}
);
});
}

@Sherrrylst
Face API – Posting data
Source code
async runFaceDetect() {
this.dataSource = [];
const imageAsset = await takePicture(
{
width: 300,
height: 500,
keepAspectRatio: true,
saveToGallery: false
});
const filePath = await this.getFilePath(imageAsset);
const imageFile = File.fromPath(filePath);
this.pictureFromCamera = imageAsset;
this.result = await this.sendRequest(imageFile);
this.dataSource = getEmotionData(this.result);
}

Azure Cognitive Services re-cap
@Sherrrylst
• Easy to use (REST API)
• Easy to provision
• Provide pre-trained models
• Possibility to use custom data
• Free training resources
• Detailed documentation
• Create account for free

@Sherrrylst
Thank you!
https://aka.ms/nativescriptday-amsterdam

Resources
• Microsoft Azure Cognitive Services: The Big Picture
• The Mojifier (MS Learn)
• The Mojifier (Github)
• Azure Cognitive Services API — I need your clothes, boots and your motorcycle
• Add conversational intelligence to your apps by using LUIS (MS Learn)
• Discover sentiment in text with the Text Analytics API (MS Learn)
• Create Intelligent Bots with the Azure Bot Service (MS Learn)
• Getting started with Azure Cognitive Services

AI Services
Cognitive Services Bot Service Azure ML, Databricks, HDInsight
Pre-build AI Conversational AI Custom AI
Azure Infrastructure
CPU, FPGA, GPU
Cosmos
DB
SQL DB Data Lake IOT EdgeDSVMSparkBatch AISQL DW

Deep learning Frameworks
Cognitive Toolkits Tensorflow Azure ML, Databricks, HDInsight
Pre-build AI Conversational AI Custom AI
Deep learning Frameworks
CPU, FPGA, GPU
Cosmos
DB
SQL DB Data Lake IOT EdgeDSVMSparkBatch AISQL DW

Using Azure Cognitive Services with NativeScript

Recommandé

Recommandé

Contenu connexe

Similaire à Using Azure Cognitive Services with NativeScript

Similaire à Using Azure Cognitive Services with NativeScript (20)

Dernier

Dernier (20)

Using Azure Cognitive Services with NativeScript

Notes de l'éditeur