Multiplaying using Networked-Aframe and Mediapipe

Hey, we have made this multiplaying 3D environment where avatars are represented by their web camera streams where the background is removed. It is meant to be used for realy cheap virtual productions. Only a tablet or laptop or mobile phone. We have used NetworkedAframe and Mediapipe Selfie Segmentation @ javascript API.


Networked-Aframe = (Aframe + EasyOpenRTC) is a library for multiplaying in VR
Mediapipe Selfie-Segmentation = GoogleAI  at WebAssembly for removing the background of the webcamera stream.
Easy-OpenRTC = an abstraction of WebRTC for p2p connections. Very handy in opposite of Janus adapter.
WebRTC couldn't transfer transparent pixels (they are automatically become black) so I placed green pixels to be transferred (black can not be easily removed).

I have made my custom shader at client side that replaces highly green pixels to the transparent ones (which means that if you wear a green blouse, then only your head will be visible).

Requirements: At least two players (it is p2p based on WebRTC)
Browser: Firefox or Chrome (in Firefox is more stable)
Device: A GPU is recommended but it runs also without it.
Permissions: It requires Mic and Camera permissions (if you  delay to give permissions, it will start without them and your avatar will be a black panel, remedy: open a new window and close the old one, do not press restart)

There are more developments. Here is my fork of Networked-Aframe : https://github.com/jimver04/networked...      See also the discussion for merging into main: https://github.com/networked-aframe/n...

• #science_technology
• #mediapipe
• #selfie_segmentation
• #javascript
• #webxr
• #networked_aframe
• #aframe
• #threejs
• #easy_openrtc