Flutterbodydetection - A flutter plugin that uses MLKit on iOS/Android platforms to enable body pose and mask detection using Pose Detection and Selfie Segmentation APIs for both static images and live camera stream.

Overview

body_detection

A flutter plugin that uses MLKit on iOS/Android platforms to enable body pose and mask detection using Pose Detection and Selfie Segmentation APIs for both static images and live camera stream. When using live camera detection it runs both the camera image aquisition and detectors on the native side, which makes it faster than using the flutter camera plugin separately.

Features

  • Body pose detection.
  • Body mask detection (selfie segmentation).
  • Run on single image or live camera feed.
  • Both camera image aquisition and detectors run together on the native side which makes it faster than using separate plugins for both and passing data between them.
  • Uses MLKit for detectors so shares its advantages and disadvantages.

Installation

iOS

Minimum required iOS version is 10.0, so if compiling for a lower target make sure to check programatically the iOS version before calling this library's functions.

To be able to use the camera-based detection you need to add a row to the ios/Runner/Info.plist file using Xcode:

  • Privacy - Camera Usage Description and set a usage description message.

If editing the file in plain-text add the following:

<key>NSCameraUsageDescription</key>
<string>Can I use the camera please?</string>

Android

Set the minimum Android SDK version to 21 (or higher) in your android/app/build.gradle file:

android {
    defaultConfig {
        minSdkVersion 21
    }
}

To use the camera you need to add uses-permission declaration to your android/app/src/main/AndroidManifest.xml manifest file:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="YOUR_PACKAGE_NAME">
  <uses-permission android:name="android.permission.CAMERA"/>
  ...

Usage

The plugin has two modes, you can either run a pose or body mask detection on a single image, or start a camera stream and enable or disable either of the detectors while it's running.

Single Image Detection

The simplest use case is to detect pose or body mask on a single image. The plugin accepts a PNG-encoded image and sends it to the native side, where it uses Google MLKit's Vision APIs to do the hard work. After that, the results are passed back to flutter:

import 'package:body_detection/body_detection.dart';
import 'package:body_detection/png_image.dart';

PngImage pngImage = PngImage.from(bytes, width: width, height: height);
final pose = await BodyDetection.detectPose(image: pngImage);
final bodyMask = await BodyDetection.detectBodyMask(image: pngImage);

This plugin provides an extension to flutter's Image widget that converts it to the PNG format so you can use any image source that the widget supports as an input: local asset, network, memory, etc.

import 'package:body_detection/body_detection.dart';
import 'package:body_detection/png_image.dart';
import 'package:flutter/widgets.dart';

void detectImagePose(Image source) async {
  PngImage? pngImage = await source.toPngImage();
  if (pngImage == null) return;
  final pose = await BodyDetection.detectPose(image: pngImage);
  ...
}

If you need the size of the encoded PNG image for your layout purposes (for example the image aspect ratio), you can get that from the returned PngImage object:

import 'package:body_detection/png_image.dart';
import 'package:flutter/widgets.dart';

Image source;
PngImage? pngImage = await source.toPngImage();
final imageSize = pngImage != null
  ? Size(pngImage!.width.toDouble(), pngImage!.height.toDouble())
  : Size.zero;

Pose object returned from the BodyDetection.detectPose call holds a list of landmarks returned from the MLKit detector. Each Landmark has a structure that follows that established by MLKit:

final pose = await BodyDetection.detectPose(image: pngImage);
for (final landmark in pose!.landmarks) {
  // The detector estimate of how likely it is that the landmark is within the image's frame.
  double inFrameLikelihood = landmark.inFrameLikelihood;

  // Position of the landmark in image plane coordinates with z value estimated by the detector.
  Point3d position = landmark.position;

  // One of the 33 detectable body landmarks.
  PoseLandmarkType type = landmark.type;
}

You can draw the pose using a custom painter:

class PosePainter extends CustomPainter {
  PosePainter({
    required this.pose,
    required this.imageSize,
  });

  final Pose pose;
  final Size imageSize;
  final circlePaint = Paint()..color = const Color.fromRGBO(0, 255, 0, 0.8);
  final linePaint = Paint()
    ..color = const Color.fromRGBO(255, 0, 0, 0.8)
    ..strokeWidth = 2;

  @override
  void paint(Canvas canvas, Size size) {
    final double hRatio =
        imageSize.width == 0 ? 1 : size.width / imageSize.width;
    final double vRatio =
        imageSize.height == 0 ? 1 : size.height / imageSize.height;

    offsetForPart(PoseLandmark part) =>
        Offset(part.position.x * hRatio, part.position.y * vRatio);

    for (final part in pose.landmarks) {
      // Draw a circular indicator for the landmark.
      canvas.drawCircle(offsetForPart(part), 5, circlePaint);

      // Draw text label for the landmark.
      TextSpan span = TextSpan(
        text: part.type.toString().substring(16),
        style: const TextStyle(
          color: Color.fromRGBO(0, 128, 255, 1),
          fontSize: 10,
        ),
      );
      TextPainter tp = TextPainter(text: span, textAlign: TextAlign.left);
      tp.textDirection = TextDirection.ltr;
      tp.layout();
      tp.paint(canvas, offsetForPart(part));
    }

    // Draw connections between the landmarks.
    final landmarksByType = {for (final it in pose.landmarks) it.type: it};
    for (final connection in connections) {
      final point1 = offsetForPart(landmarksByType[connection[0]]!);
      final point2 = offsetForPart(landmarksByType[connection[1]]!);
      canvas.drawLine(point1, point2, linePaint);
    }
  }

  ...
}

And use that in your widget tree:

@override
Widget build(BuildContext context) {
  // Use ClipRect so that custom painter doesn't draw outside of the widget area.
  return ClipRect(
    child: CustomPaint(
      child: _sourceImage,
      foregroundPainter: PosePainter(
        pose: _detectedPose,
        imageSize: _imageSize,
      ),
    ),
  );
}

For details see Google MLKit Pose Detection API documentation.

Detected body mask is returned as a double-valued buffer of length width * height indicating the confidence that a particular pixel covers area that is recognized to be a body. Size of the body mask can differ from the size of the input image as this makes calculations faster and makes it possible to run acceptably well in real-time on slower devices as well.

To display the mask you can for example decode the buffer as a dart image using the buffer value as an alpha component:

final mask = await BodyDetection.detectBodyMask(image: pngImage);
final bytes = mask.buffer
    .expand((it) => [0, 0, 0, (it * 255).toInt()])
    .toList();
ui.decodeImageFromPixels(Uint8List.fromList(bytes), mask.width, mask.height, ui.PixelFormat.rgba8888, (image) {
  // Do something with the image, for example set it as a widget's state field so you can pass it to a custom painter for drawing in the build method.
});

Then for example to draw transparent blue overlay over the background you could have a custom painter like this:

class MaskPainter extends CustomPainter {
  MaskPainter({
    required this.mask,
  });

  final ui.Image mask;
  final maskPaint = Paint()
    ..colorFilter = const ColorFilter.mode(
        Color.fromRGBO(0, 0, 255, 0.5), BlendMode.srcOut);

  @override
  void paint(Canvas canvas, Size size) {
    canvas.drawImageRect(
        mask,
        Rect.fromLTWH(0, 0, mask.width.toDouble(), mask.height.toDouble()),
        Rect.fromLTWH(0, 0, size.width, size.height),
        maskPaint);
  }
}

And use that in your widget tree together with original image:

@override
Widget build(BuildContext context) {
  return CustomPaint(
    child: _sourceImage,
    foregroundPainter: MaskPainter(mask: _maskImage),
  );

For details see Google MLKit Selfie Segmentation API documentation.

You can see the complete example in the plugin's repository.

Camera Feed Detection

In addition to single image detection, this plugin supports real-time detection of either or both the pose and body mask from camera feed. Both the camera image acquisition and detection run on the native side, so we eliminate the cost of data serialization we would need if we used a separate flutter plugins for each.

To start and stop the camera stream, use the following methods while passing callbacks for when the camera frame or detection results are ready:

await BodyDetection.startCameraStream(
  onFrameAvailable: (ImageResult image) {},
  onPoseAvailable: (Pose? pose) {},
  onMaskAvailable: (BodyMask? mask) {},
);
await BodyDetection.stopCameraStream();

Detectors are disabled by default. To enable or disable particular detector use the following methods:

await BodyDetection.enablePoseDetection();
await BodyDetection.disablePoseDetection();

await BodyDetection.enableBodyMaskDetection();
await BodyDetection.disableBodyMaskDetection();

You can run those both before starting the camera stream as well as while the stream is already running.

To display camera image you can create an Image widget from bytes:

void _handleCameraImage(ImageResult result) {
  // Ignore callback if navigated out of the page.
  if (!mounted) return;

  // To avoid a memory leak issue.
  // https://github.com/flutter/flutter/issues/60160
  PaintingBinding.instance?.imageCache?.clear();
  PaintingBinding.instance?.imageCache?.clearLiveImages();

  final image = Image.memory(
    result.bytes,
    gaplessPlayback: true,
    fit: BoxFit.contain,
  );

  setState(() {
    _cameraImage = image;
    _imageSize = result.size;
  });
}

ImageResult's bytes field contains a JPEG-encoded frame from the camera which Image.memory() factory constructor accepts as its input.

Example App

Running Pose Detection

Demo

Running Body Mask Detection

Demo

Future Developments

This library encodes images from the camera and serializes them when passing them over to flutter. This enables easy access to the image data so one can do additional processing, save frames to storage or send them for over the network, but also adds an overhead which could be avoided if it only passed a texture id which could then be displayed using the Texture widget in the same vein as what camera plugin does.

It does not yet provide any configuration options and rely on defaults ("accurate" pose detection model, raw size mask for selfie segmentation, using first available front camera, default camera image size, JPEG encoding quality set to 60). It would be good to let users choose those settings to match their use case.

Exception handling is not unified and could use some more work.

Notice

This library is not yet battle-tested. As it's an early release it might have its features and API changed. If you encounter a problem or would like to request a feature, feel free to open an issue at the project's GitHub page.

MLKit's Pose Detection and Selfie Segmentation are still in beta and hence this software should also be considered as such.

You might also like...

Klutter plugin makes it possible to write a Flutter plugin for both Android and iOS using Kotlin only.

The Klutter Framework makes it possible to write a Flutter plugin for both Android and iOS using Kotlin Multiplatform. Instead of writing platform spe

Sep 13, 2022

WooCommerce App template that uses Flutter. Integrated to work with WooCommerce stores, connect and create an IOS and Android app from Flutter for IOS and Android

WooCommerce App template that uses Flutter. Integrated to work with WooCommerce stores, connect and create an IOS and Android app from Flutter for IOS and Android

WooCommerce App: Label StoreMax Label StoreMax - v5.3.1 Official WooSignal WooCommerce App About Label StoreMax Label StoreMax is an App Template for

Sep 22, 2022

Flutter Local Notifications - Learn how to implement local notifications into both Android and iOS using flutter_local_notifications plugin.

Flutter Local Notifications - Learn how to implement local notifications into both Android and iOS using flutter_local_notifications plugin.

Flutter Local Notifications Example Flutter Local Notifications - Learn how to implement local notifications into both Android and iOS using flutter_l

Aug 4, 2022

An app to pick, upload and display images from camera and gallery with size and extension constraints.

image_uploader A new Flutter project. Getting Started This project is a starting point for a Flutter application. A few resources to get you started i

Mar 7, 2022

A powerful plugin that fully uses the native image library's ability to display images on the flutter side.

PowerImage A powerful plugin that fully uses the native image library's ability to display images on the flutter side. 中文文档 Features: Supports the abi

Sep 21, 2022

A Video Player For Vimeo Videos in Flutter. This plugin allows us to play video from Vimeo and it supports Android and iOS platforms.

A Video Player For Vimeo Videos in Flutter. This plugin allows us to play video from Vimeo and it supports Android and iOS platforms.

vimeo_video_player A Video Player For Vimeo Videos in Flutter. This plugin allow us to play video from vimeo and it's supports Android and iOS platfor

Sep 5, 2022

A Flutter plugin that provides assets abstraction management APIs without UI integration, you can get assets (image/video/audio) on Android, iOS and macOS.

A Flutter plugin that provides assets abstraction management APIs without UI integration, you can get assets (image/video/audio) on Android, iOS and macOS.

photo_manager Photo/Assets management APIs for Flutter without UI integration, you can get assets (image/video/audio) from Android, iOS and macOS. 提供相

Sep 23, 2022

A Flutter plugin to get location updates in the background for both Android and iOS

Background Location A Flutter plugin to get location updates in the background for both Android and iOS (Requires iOS 10.0+). Uses CoreLocation for iO

Sep 22, 2022

A server app built using Shelf, configured to enable running with Docker

A server app built using Shelf, configured to enable running with Docker

A server app built using Shelf, configured to enable running with Docker. This sample code handles HTTP GET requests to / and /echo/message Requests

Jan 26, 2022
Comments
  • Toggling camera for body detection

    Toggling camera for body detection

    I'm enjoying this plugin so far. It appears the camera selection is stored in BodyDetection.startCameraStream. Is there a way to easily toggle front versus back camera within the app? Thanks

    opened by dcwhitt 0
  • Fatal Exeption on low end device

    Fatal Exeption on low end device

    Thank you for your help and answers. On lowend device I stil get the exeption I/CameraManagerGlobal(10088): Connecting to camera service D/CameraRepository(10088): Added camera: 0 I/Camera2CameraInfo(10088): Device Level: INFO_SUPPORTED_HARDWARE_LEVEL_LEGACY D/UseCaseAttachState(10088): Active and attached use case: [] for camera: 0 D/AndroidRuntime(10088): Shutting down VM E/AndroidRuntime(10088): FATAL EXCEPTION: main E/AndroidRuntime(10088): Process: com.u0x48lab.body_detection_example, PID: 10088 E/AndroidRuntime(10088): java.lang.IllegalArgumentException: No available camera can be found. E/AndroidRuntime(10088): at androidx.camera.core.CameraSelector.filter(CameraSelector.java:107) E/AndroidRuntime(10088): at androidx.camera.core.CameraSelector.filter(CameraSelector.java:137) E/AndroidRuntime(10088): at androidx.camera.lifecycle.ProcessCameraProvider.bindToLifecycle(ProcessCameraProvider.java:388) E/AndroidRuntime(10088): at androidx.camera.lifecycle.ProcessCameraProvider.bindToLifecycle(ProcessCameraProvider.java:274) E/AndroidRuntime(10088): at com.u0x48lab.body_detection.CameraSession.bindAnalysisUseCase(CameraSession.kt:120) E/AndroidRuntime(10088): at com.u0x48lab.body_detection.CameraSession.access$bindAnalysisUseCase(CameraSession.kt:17) E/AndroidRuntime(10088): at com.u0x48lab.body_detection.CameraSession$1.run(CameraSession.kt:32) E/AndroidRuntime(10088): at android.os.Handler.handleCallback(Handler.java:790) E/AndroidRuntime(10088): at android.os.Handler.dispatchMessage(Handler.java:99) E/AndroidRuntime(10088): at android.os.Looper.loop(Looper.java:164) E/AndroidRuntime(10088): at android.app.ActivityThread.main(ActivityThread.java:6523) E/AndroidRuntime(10088): at java.lang.reflect.Method.invoke(Native Method) E/AndroidRuntime(10088): at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:438) E/AndroidRuntime(10088): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:857)

    opened by JackJagt 8
Owner
null
Face Mask Detection with Tensorflow(Flutter)Face Mask Detection with Tensorflow(Flutter)

Face Mask Detection with Tensorflow(Flutter) Face Mask Detection With TFlite Info An app made with flutter and tensor flow lite for face mask detectio

Mohsen Mirdar 1 May 23, 2022
An rx stream builder widget that is able to pre-populate a flutter StreamBuilder with data from an rx stream if the stream is either a value or a replay observable.

An rx stream builder widget that is able to pre-populate a flutter StreamBuilder with data from an rx stream if the stream is either a value or a replay observable. For example the RX stream is a BehaviorSubject or a ReplaySubject.

Jon Samwell 8 Jan 22, 2022
📲 Flutter application that uses CycleGAN to generates an anime version from a selfie.

This application allows you to capture a picture of yourself or upload an existing image, then using Cycle-Consistent Adversarial Network (CycleGAN) t

null 3 Jun 19, 2022
Flutter plugin for selecting images from the Android and iOS image library, taking new pictures with the camera, and edit them before using such as rotation, cropping, adding sticker/text/filters.

advance_image_picker Flutter plugin for selecting multiple images from the Android and iOS image library, taking new pictures with the camera, and edi

Weta Vietnam 89 Sep 22, 2022
My flutter (android, ios) UI design examples 🎈 - user profile UIs, food order ui, splashscreen, mask widget usage, settings page ui

Flutter UI Design Examples ?? This repository contains the flutter ui designs I designed while learning. Doctor Appointment App UI Packages in use: fl

Aleyna Eser 22 Sep 26, 2022
This is a flutter plugin to detect edges in a live camera, take the picture of detected edges object, crop it, and save.

edge_detection A flutter plugin to detect edges of objects, scan paper, detect corners, detect rectangles. It allows cropping of the detected object i

Sawan Kumar Bundelkhandi 170 Sep 18, 2022
Real-time object detection in Flutter using camera and tflite plugin

For details: https://medium.com/@shaqian629/real-time-object-detection-in-flutter-b31c7ff9ef96 flutter_realtime_detection Real-time object detection i

Post_Swift 5 Jul 21, 2022
Body-Mass-Index-Calculator - Body Mass Index Calculator Using Flutter

Body-Mass-Index-Calculator Image Video Presentation bandicam.2022-02-18.00-33-22

Harsh Vardhan 3 Aug 15, 2022
A full-fledged camera app built with Flutter using the camera package.

Flutter Camera Demo A full-fledged camera app built with Flutter using the camera package. You can even add custom features to this app and customize

NguyenHoangDuc 5 Aug 19, 2022
This is an application that uses the Flutter framework, SQFLite as a database to record blood pressure, blood sugar, BMI, or create medication reminders in multi mobile platforms You can run this project on iOS, Android

This is an application that uses the Flutter framework, SQFLite as a database to record blood pressure, blood sugar, BMI, or create medication reminders in multi mobile platforms You can run this project on iOS, Android

null 10 Aug 26, 2022