一、WWDC2018 Vision
去年IOS11出了Vision框架给开发者提供了使用简单的图像识别方式，本来期待在今年能够拥有更多的图像处理的功能，但是从WWDC2018看来，苹果此番针对Vision框架并没有进行大幅度的升级，功能未变，只是针对IOS12有增加一些修订含义的常量，比如：
- VNDetectFaceLandmarksRequestRevision1
- VNDetectFaceLandmarksRequestRevision2
- VNDetectHorizonRequestRevision1
而关于Vision框架的使用只有两个session的讲解，分别是两个场景下的使用：
- Vision with Core ML
- Object Tracking in Vision
场景的使用下的使用并不复杂，我们通过一个具体的Demo来看看。
二、Vision调用CoreML
苹果在大会上演示了一个Demo,Vision框架通过调用CoreML在相机实时的视频流检测识别出物体名称,我们这里也来实现一个。
1、通过AVFoundation构建一个相机
```- (void)initAVCapturWritterConfig{ self.session = [[AVCaptureSession alloc] init]; //视频 AVCaptureDevice *videoDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo]; if (videoDevice.isFocusPointOfInterestSupported && [videoDevice isFocusModeSupported:AVCaptureFocusModeContinuousAutoFocus]) { [videoDevice lockForConfiguration:nil]; [videoDevice setFocusMode:AVCaptureFocusModeContinuousAutoFocus]; [videoDevice unlockForConfiguration]; } AVCaptureDeviceInput *cameraDeviceInput = [[AVCaptureDeviceInput alloc] initWithDevice:videoDevice error:nil]; if ([self.session canAddInput:cameraDeviceInput]) { [self.session addInput:cameraDeviceInput]; } //视频 self.videoOutPut = [[AVCaptureVideoDataOutput alloc] init]; NSDictionary * outputSettings = [[NSDictionary alloc] initWithObjectsAndKeys:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA],(id)kCVPixelBufferPixelFormatTypeKey, nil]; [self.videoOutPut setVideoSettings:outputSettings]; if ([self.session canAddOutput:self.videoOutPut]) { [self.session addOutput:self.videoOutPut]; } self.videoConnection = [self.videoOutPut connectionWithMediaType:AVMediaTypeVideo]; self.videoConnection.enabled = NO; [self.videoConnection setVideoOrientation:AVCaptureVideoOrientationPortrait]; //初始化预览图层 self.previewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:self.session]; [self.previewLayer setVideoGravity:AVLayerVideoGravityResizeAspectFill];}``` 2、引入CoreML的模型
3、初始化Vision框架的请求
``` //实物识别 VNCoreMLModel *vnModel = [VNCoreMLModel modelForMLModel:[MobileNet new].model error:nil]; self.coreMLRequest = [[VNCoreMLRequest alloc] initWithModel:vnModel completionHandler:^(VNRequest * _Nonnull request, NSError * _Nullable error) { VNCoreMLRequest *coreR = (VNCoreMLRequest *)request; VNClassificationObservation *firstObservation = [coreR.results firstObject]; dispatch_async(dispatch_get_main_queue(), ^{ if (firstObservation) { self.googleLabel.text = firstObservation.identifier; } else { self.googleLabel.text = @""; } }); }]; self.coreMLRequest.imageCropAndScaleOption = VNImageCropAndScaleOptionCenterCrop;```4、相机回调执行
```- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection{ UIImage *image = [UIImage imageFromSampleBuffer:sampleBuffer]; UIImage *scaledImage = [image scaleToSize:CGSizeMake(224, 224)]; CVPixelBufferRef buffer = [image pixelBufferFromCGImage:scaledImage]; VNImageRequestHandler *handler = [[VNImageRequestHandler alloc] initWithCVPixelBuffer:buffer options:@{}]; NSError *error; [handler performRequests:@[self.coreMLRequest] error:&error];}```5、结果展示
当我获取的画面返回的时候就会通过MobileNet这个机器学习模型去识别，结果展示在左下角的标签里面。这样也就完成了Vision在CoreML上的调用。
三、Vision实现物体追踪
1、人脸请求
这里我们没有使用VNTrackObjectRequest，这里使用了VNDetectFaceLandmarksRequest来实现一个脸部追踪贴纸的效果，调用是一样的。
在上面相机的基础上，我们新建一个脸部识别的请求
``` self.faceRequest = [[VNDetectFaceLandmarksRequest alloc] initWithCompletionHandler:^(VNRequest * _Nonnull request, NSError * _Nullable error) { VNDetectFaceLandmarksRequest *faceRequest = (VNDetectFaceLandmarksRequest*)request; VNFaceObservation *firstObservation = [faceRequest.results firstObject]; dispatch_async(dispatch_get_main_queue(), ^{ if (firstObservation) { CGRect boundingBox = [firstObservation boundingBox]; CGRect rect = VNImageRectForNormalizedRect(boundingBox,self.realTimeView.frame.size.width,self.realTimeView.frame.size.height); CGRect frame = CGRectMake(self.realTimeView.frame.size.width - rect.origin.x - rect.size.width, self.realTimeView.frame.size.height - rect.origin.y - rect.size.height, rect.size.width, rect.size.height); self.maskView.frame = frame; self.maskView.hidden = NO; } else { self.maskView.hidden = YES; } }) }];```2、相机回调切换
在上面相机回调的基础上增加一个按钮切换请求模式即可
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection { if (self.coreMlMode) { UIImage *image = [UIImage imageFromSampleBuffer:sampleBuffer]; UIImage *scaledImage = [image scaleToSize:CGSizeMake(224, 224)]; CVPixelBufferRef buffer = [image pixelBufferFromCGImage:scaledImage]; VNImageRequestHandler *handler = [[VNImageRequestHandler alloc] initWithCVPixelBuffer:buffer options:@{}]; NSError *error; [handler performRequests:@[self.coreMLRequest] error:&error]; } else { CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer); VNImageRequestHandler *handler = [[VNImageRequestHandler alloc] initWithCVPixelBuffer:(CVPixelBufferRef)imageBuffer options:@{}]; NSError *error; [handler performRequests:@[self.faceRequest] error:&error]; } }3、结果展示
我们把镜头放在同事的脸上，就会识别出同事的脸部位置，将预先插入的maskView的frame设置在对应的位置，就能让面具一直追踪脸部紧贴，当没有识别出脸部的时候，就会隐藏面具，效果如下。
四、小结
Vision框架为我们封装的视觉处理一些场景下的功能，调用非常简单，但是正是由于调用的简单，对应就达不到一个复杂的功能，一般场景是可以实现的，期待苹果未来能够提供更为丰富的API，比如图片的风格变换等等，我们的应用也会越来越丰富。
分享到:
閱讀更多 今日頭條技術團隊 的文章
