Authorize MIM Portal user image upload with Microsoft Cognitive Services


I saw these great videos from //build keynote some weeks ago about the Microsoft Cognitive Services and I was really impressed. I know these APIs like face, emotion, speech are designed for other purposes but I was thinking to myself on who to benefit from them for identity management.

So I remembered some time ago when talking about MIM Portal as a user self-service portal for personal data some customers find it is sometimes not a good idea if users can upload their own photos. The arguments where that photos cannot be validated in that way that it really belongs to that person. So people could upload for example funny pictures and avatars or even more bad images.

Sure, you can handle this by organizational policies, but I was thinking of a technical solution. At this point when thinking about Microsoft Cognitive Services the Face API came to my mind.

If you check the Face API it has methods for face detection in images and also face identification or verifying. You can also create person groups and persons with multiple faces saved in Azure if you want.

But for my little demo I only need the face detect and verify methods.

So here is how this demo works:

If people upload new images to MIM Portal, I trigger an authorization workflow and get the current and new photo with an MIMWAL update resource activity and pass that data to a PowerShell script which then calls the Face API.

The PowerShell Script uploads both images to Azure to do a face detection within the image and then returns a faceId for each of the pictures. Images are saved 24 hours Azure.

With the faceId’s I call the verify method of the Face API and if the person on both images are identical it returns a JSON object with a Boolean value “isIdentical” and a confidence.

If isIdentical is not True I throw an error to have the workflow getting a denied in MIM service.

The Microsoft Cognitive Services are still preview but I have no issues with verifying photo of mine which are pretty new and some are over 20 years old.

Note:
For this demo to work there must already a user photo in place.

 

So here is what you need to do to set this up in your environment:

Create an account for Microsoft Cognitive Services, you can either go to the Cognitive Services page and sign-up for an account where you can request trial keys to use the services.

CognitiveService1

If you have an Azure subscription (MSDN, Trial, Paid) you can to the new Azure Portal click “Browse >” on the menu and search for Microsoft Cognitive Services.

Create a Cognitive Service account with Face API and the Free Tier like I did (pretty easy), on the Settings blade you can get the Face API Key you will need to enter in the script later on.

CognitiveService2

Now it is time for the MIM parts of the solution. We need to create a workflow with 2 activities like following:

Create a workflow “Verify person image with cognitive services”

Activity 1:

Is a MIMWAL update resource activity (You can possibly use the function evaluator here) which passes to values to the workflow dictionary:

  • [//Target/Photo] -> [//WorkflowData/OldPhoto]
  • [//Request/RequestParameter] -> [//WorkflowData/AllChanges]

CognitiveService3

 

Activity 2:

Is a PowerShell workflow activity, you can either use the MIMWAL PowerShell or the one from Codeplex. There are 2 scripts, one that is entered in the activity, and another one in the file system that is called from the first script.

Put the following script directly in the PowerShell activity:

# Get Changes (new photo) and old photo from workflow data
$ChangeList = $fimwf.WorkflowDictionary.AllChanges
$OldPhoto = $fimwf.WorkflowDictionary.OldPhoto

# Get GUID of current request for temp file creation
$RequestId = $fimwf.RequestId.Guid

# Configuration parameters, modify to your needs
$VerifyScriptPath = "C:\Temp\Pictures\VerifyImage.ps1"
$TempFolder = "C:\temp\Pictures\"
$TempFileOld = $TempFolder + $RequestId + "_old.jpg"
$TempFileNew = $TempFolder + $RequestId + "_new.jpg"

# Check als changes of current request and save new photo to temp image file
Foreach ($Change in $ChangeList)
{
$ChangeXML = [XML]$Change
If ($ChangeXML.RequestParameter.PropertyName -eq "Photo")
{
$Image = $ChangeXML.RequestParameter.Value.'#text'
[System.Convert]::FromBase64String($Image) | Set-Content $TempFileNew -Encoding Byte
}
}

# Save old photo to temp image file
$OldPhoto | Set-Content $TempFileOld -Encoding Byte

# Call Cognitive Services verification script (Cannot run in PS Version 2)
$resultstring = powershell -Version 4.0 -File $VerifyScriptPath -OldPhoto $TempFileOld -NewPhoto $TempFileNew
$result=[System.Convert]::ToBoolean($resultstring)

# Remove temporary image files
Remove-Item $TempFileOld -Force
Remove-Item $TempFileNew -Force

# Throw an error if person shown in images is not identical to deny the request
if (-not $result)
{
throw "Image verify failed: Picture does not show the user."
}

Check the “Configuration parameters” section of the script and modify it to your needs.

Put the following script to the file system, default is C:\Temp\Pictures\VerifyImage.ps1:

# Face API
param($OldPhoto,$NewPhoto)

# Set FaceAPI Key of your Cognitive Services Subscription
$FaceApiKey = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxx'

# Function for detecting faces in images, returns faceId for use in VerfiyPerson
function DetectFace([string]$ImageFilePath)
{
# Set request header for API call
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Content-Type", 'application/octet-stream')
$headers.Add("Ocp-Apim-Subscription-Key", $FaceApiKey)

# Get byte-array from image file
[byte[]]$body = Get-Content $ImageFilePath -Encoding Byte -ReadCount 0

# Call Microsoft Cognitive Services Face API
$response = Invoke-RestMethod 'https://api.projectoxford.ai/face/v1.0/detect?returnFaceId=true' -Headers $headers -Body $body -Method Post

# Return JSON data of faceIds
ConvertTo-Json $response
}

# Function for verifing to faces, returns if 2 faces are from an identical person
function VerifyPerson([string]$faceId1,[string]$faceId2)
{
# Set request header for API call
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Content-Type", 'application/json')
$headers.Add("Ocp-Apim-Subscription-Key", $FaceApiKey)

# Create JSON request body for input to API
[string]$body=(New-Object PSObject |
Add-Member -PassThru NoteProperty faceId1 $faceId1 |
Add-Member -PassThru NoteProperty faceId2 $faceId2
) | ConvertTo-JSON

# Call Microsoft Cognitive Services Face API
$response = Invoke-RestMethod 'https://api.projectoxford.ai/face/v1.0/verify' -Headers $headers -Body $body -Method Post

# Return JSON data of verification request
ConvertTo-Json $response
}

# Detect faces of old and new MIM Portal user images
$faceJson1 = DetectFace -ImageFilePath $OldPhoto
$faceJson2 = DetectFace -ImageFilePath $NewPhoto

# Get faceIds from JSON data
$faceId1 = (ConvertFrom-Json $faceJson1).value.faceId
$faceId2 = (ConvertFrom-Json $faceJson2).value.faceId

# Verify old and new face, returns an true if person is identical
$verify = VerifyPerson -faceId1 $faceId1 -faceId2 $faceId2 | ConvertFrom-Json

# Pass back result to calling script
$verify.IsIdentical

Enter your Face API Key of your subscription into the FaceAPIKey Variable.

 

Last thing is to create an MPR (Management Policy Rule) to trigger our workflow. As a simple example you can create it like this:

  • MPR: Verify person photo on new image upload
  • Policy Type: Request based
  • Requestor: All People
  • Operation: Modify single value attribute
  • Target Resource Before/After: All People
  • Resource Attribute: Photo
  • Authorization Workflow: {The workflow created above}

Done…You can now upload a new photo for a user and it will only be valid if the persons on the new photo is identical with the old one. Depending on the speed of detection/verification you will see a “Pending approval” message when you submit the new photo. Check the request history on completition.

Here is an example of a denied request because I uploaded a photo from my collegue:

CognitiveService4

 

Advertisements

About Peter Stapf
Senior Consultant Identity and Access MVP (Enterprise Mobility)

3 Responses to Authorize MIM Portal user image upload with Microsoft Cognitive Services

  1. tlkidentity says:

    Great post Peter. very interesting. In line with a tool I am working on, Voice control of FIM/MIM.

  2. Peter Stapf says:

    That was something that also came to my mind Ike, maybe with some use of Bot Framework or the speech APIs. My tought is have a voice approval, or some communication with Skype.

    So just saying “Yes” to an approval could be a funny solution, regardless if someone uses this really in production.

  3. tlkidentity says:

    Peter, the speech API is exactly what i am looking for. I am looking at control of MIM processes via Voice. Like “Computer, build a synchronization rule between for MA X”. And access to MIM will be via Speech, disregard the AD security. The big question for me though is how do I build some fun stuff into MIM via speech. I have some ideas I am harvesting. More to come. Thanks again for sharing cognitive APIs!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: