Lei feng’s network: the author, Ai Tao, easy Apple CTO, graduated in computer engineering from the University of Toronto. For many years following the “father of the world wearable computers,” Professor Steve Mann, the basic research focus on the smart glasses. Recently, Apple’s core product VMG-PROV announces open source geeks, this video perspective of mediated reality glasses, why hasn’t officially launched open source? Why choose video perspective? Author to his personal experience to answer the question-“when we call VMG-PROV after the open source, we hope to expose all the problems there are products. “With the exception of talk about the pros and cons of the product itself, this is a serious article.
Following is the author bio, Lei feng’s network (search for “Lei feng’s network”, public interest) do not adjust the intent of editing.
Today I write a small article about VMG-PROV (the so-called open source geek Edition) what exactly is a thing, friends of hope reading this thing is going to take it in hand to yourself “toys” have some knowledge. Of course, the local casual, I think VMG-RPOV stand home are pretty smart.
| A source of blood
Remember me to mentor Steve Mann’s lab, see a costume freak engineers (including one with no tops hair man) is a pile of “destroying the old” electronic devices, monitors, and surrounded by various electronic components, while they are using the interface quite Chueh-Li’s unknown alternative program to do some things. That Flash was excited at the same time, also felt a little embarrassed for their ignorance.
“Hi Steve, what software are they using?” With curiosity I asked, feeling weak burst.
“Oh Arkin, I can’t answer that if it’s not the right question.” Professor no shoes, with a soldering iron in hand, head back to me said. Maybe he doesn’t know I’m petrochemical.
“You know, Arkin, we don’t use any software in this Lab.” Professor down the iron, looking over towards me.
“Software are programs that you need to pay. Paying for software that you can’t hack or share is ridiculous.” Professor suddenly serious again. But I also understand why these programs are so strange.
I nodded, muses: “So everything here is open-source? Like everything?”
Professor smiled and nodded, “said Yes,Everything.”
This time working in the lab, I was exposed to a wide variety of open source tools. Professor said was true, in the laboratory, our operating system is Ubuntu instead of Windows or MacOS; image editing or Photoshop instead of GIMP; even programs like Word and Excel, we used LibreOffice instead. Is more exaggerated, as researchers, we don’t have Matlab, but using a thing called Octave.
Steve used to be a work of art, in the collection now at the Austin Museum of art. In the figure below Installation called “License To Sit” is a License can only be used to pay for the Chair. Just like paid software, this Chair will quickly remind you to pay when due, otherwise, once expired, eject pin and forcibly stopped the Chair can provide you “sit” service. This work of art is fun, can make us think when rules are used in the digital world to some funny situations in the physical world.
Looking at the Chair, remembers the first time he met Professor of that conversation, I suddenly realized a long consideration decided to actually have sources, depicting all had full blood of open source.
| Sent to the geek’s hands
Recently, we decided to make our own inadequate engineering VMG-PROV as an open source hardware and dropped it into the market in the past.
The move, in my opinion, there are contributing factors there are selfish reasons.
We really solved the problem of some smart glasses. The aims of open source, I hope fellow and aspiring geek by dismantling the VMG-PROV studies, could not repeat to solve these issues. The wheel here, do not need to reinvent it. In addition, we do the smart glasses are different from VR/AR, slightly ahead of the point. In order to make the ideal products as soon as possible, small teams of 20 people we need more capable of helping. When we call VMG-PROV after the open source, we hope to expose all the problems there are products. While geeks have jointly pointing to perfecting this product and we stand on the same starting line. Open source will be the future direction of development of science and technology, a geek of some accomplishments should not think about how it’s not perfect “little secrets” to make money, ought to be thinking about how to get people involved get things done as soon as possible.
We hope to VMG-PROV is delivered to the right hands, this means that this product is not a consumer-level, or even the developer level, we call it a “geek”. We want to sell him to have open source blood, wanted to explore the digital vision, and have the capability of geeks. See here, I think readers should buy the VMG-PROV already had some little judge.
| Why choose video perspective?
VMG (included in the pre-PROV and MARK two version in development) is an open source, is a find mediated reality (and digital Visual) tools. We believe that virtual reality (VR) and augmented reality (AR) is a reflection of digital vision. Therefore, VMG first VR/AR video fluoroscopy can be compatible in a head at the same time. Wrote an article about the smart glasses before little article also mentioned that the AR time, mixed reality (Mixed Reality) or mediated reality (Mediated Reality), the video perspective is necessary to cross a threshold.
I am often asked, optical perspective programme in the market so much, and everyone was doing optical perspective, why did you have to select the video perspective? I said, I did not choose the video perspective video perspective of chose me.
Before entering the lab I majored in FPGA, and because of this, Steve let me use FPGA to conquer some GPU can’t run on the smart glasses algorithm, results, a senior and I present our findings on the SIGGRAPH exhibition. In the human brain explosion on the show, I chat with seniors in each identified future research directions. After returning to the lab, I mainly make video perspective and related software, and he began to study the optical perspective of things. Video perspective (Video See-Through, or VST) and optical perspective like HoloLens (Optical See-Through or OST), each with advantages and disadvantages. Now, I in AR applications, for example, probably sum up the two main problems. (Academia about the comparison between very thorough, we are interested to read the relevant papers. ）
Video perspective (Video See-Through, or VST) and optical perspective like HoloLens (Optical See-Through or OST)–
1, on the overlay and display view (VST victory)
Optical see through a special design will be projected onto a translucent digital screen on the display device, so show the occlusion effect (Occlusion) is not perfect. See Magic Leap earlier issued a patent for glasses you know, they spend at least 7 pages of content describing the complexity of their optical path design to reduce the occlusion. Optical see optical design is complex and would like to make a small angle effect there. Try HoloLens people (about 40 degrees) will be able to feel the small is a perspective that I feel.
Also, because of the complex design and processing difficulties, OST of the cost will be very high. However, translucent is translucent, you slow down, it is impossible to solve. In this connection, the VST win OST. View of the collection of pictures through the camera, digitized before it is displayed to the screen, blocking effect can relax at a low cost to be perfect. In this regard, VST can use algorithms to make real-time p, let the reality into your palette.
2, about the real picture delay and distortion (OST victory)
Due to the use of the camera, plus computers need to handle display and digital content, VST delay larger than the OST.
VST hardware like VR, camera and display refresh delay is congenital. On the OST, the real picture is the outside world through optical devices through inherent zero-delay and zero distortion. Video perspective in addition to refresh delay also needs correction because the position of the camera and optics and screen gap caused people like, nature will be delayed. We spend a lot of time, through the optimization of hardware and software to reduce latency. But the delay is never zero, in this connection, the OST out explosive VST. FENDI iPhone 6 Plus Case
3, on the VAC problem (tie)
Both OST and VST, VAC problems exist. This is what we use to display, depth perception mechanism in my eyes (Vergence) and monocular (Accommodation) when there is a conflict causing dizziness problem.
Interested readers can check out Vergence Accommodation Conflict or “vergence conflict.”
(Lei Feng network Note: VAC, you can view this analysis article the VAC has plagued virtual reality, is really no solution to it? 》）
Magic in the big Leap, this issue was also slowed by light field described a large number of technologies. In their prior to shipment, in this issue, current OST and VST, neck and neck.
| Geeks read on
VMG-PROV as a video perspective exploration tools, performance is limited. I hope the next description for you geeks recognize the risk of purchase PROV.
Delay in the VMG-MARK framework has greatly improved, and finally I will tell you details. Below is a screenshot from our Demo to be released open source project, a digital model can be superimposed on the desktop without preset Marker.
In the open source Demo, VMG-PROV through binocular cameras capture the real scenes in, after processing by the FPGA on the head, through the USB3. 0 interface on the transmission to the connected PC. Real scenes by software calibration, Visual SLAM algorithm starts operation, digital model will default to the correct location. And then transmitted through the HDMI screen display on the screen at the end. This open source project contains:
1.VMG-PROV hardware detail and schematic diagram
As shown in the following illustration, VMG-PROV hardware parameters have been marked and are listed in the following table. Camera video footage collected through FPGA processing incoming in the connected PC, finally SLAM the superposition eye back in the digital model of the picture. So when used on the AR, VMG-PROV PC complete larger video stimulation, so it is recommended using high desktop distribution.
As an MR head and VMG-PROV parameters can only reach VR entry level. I think to make a stunning application on the MR, this generation was not enough. That is why I hope we can take this as a learning and exploration tools, and also through your feedback to us more suggestions for improvement. In order to make this possible, we will provide a VMG-PROV diagram.
2. on the FPGA image processing hardware description language
VST is superior to the OST key point is the picture of reality changes, simply put, is through the algorithm, p.
In testing out the software algorithms to FPGA implementations or accelerated reduction of the delay is of great help. In our open source projects, contains a Demo of MR, synthesis and real-time high dynamic range image mapping (Real-Time HDR Composition and Tonal Mapping), exclusively through the logic into the FPGA inside.
The following figure, high-speed conversion exposure in FPGA control Sensor, and synthesizer to select from dark to light the best part of each picture.
FPGA is a magical chip is a chip in the transformer. FPGA engineer knows, FPGA not everyone got.
Below we are going to open source projects is given in the FPGA hardware description language in parts. Raw data from the Sensor came in would be completely ISP processing in FPGA, HDR synthesis after the USB3.0 to PC. I believe the project open source Verilog HDL, can attract members had the courage to master to join the chip-level exploration of MR. By FPGA combined with demand on the PC side, further reduce the delay VST.
3. based on the PTAM binocular SLAM source code
VMG-PROV the SLAM is based on binocular RGB camera, was changed by the famous PTAM. This works above Miku stack the glasses in a video picture. PTAM then, we will knock sensor data fusion, and using different methods (such as ORB-SLAM or LSD-SLAM) to achieve more stable stacking. I don’t SLAM this drawing, including open source content, calibration of binocular camera correction, FAST feature extraction, RANSAC and Bundle Adjustment, and the model of Miku. Need to use Unity3D friends, the project also has to import the SLAM. Also, when we went to see the source code, you can reference PTAM information modified.
| Ordinary developers wait
At this point, I think we need to do not need to purchase the “geek” has a certain amount of judgment. Since VMG-PROV has not reached the level, you want to do in VMG application needs to wait VMG-MARK (developer version). At the current pace, our research and development teams are VMG-MARK most of the technical problems have been resolved. Our hardware and software architecture is called VMG-MARK VLLV (Very Low Latency Video See-Through), it will delay video perspective problems greatly slowed (slow is not the solution, solution is Zero Latency). Therefore, I hope that a landing application developers the best use the product MARK, rather than a VMG-PROV geek version.
The schema below is VMG-MARK, initially was designed to reduce the amount of video throughput calculation of PC demand, high real scenes of lofty frame streaming video directly from the Sensor into the screen, without a computer. Zhihou, slowly evolved into four modes.
We call this framework VT Architecture, its four operating modes are:
Real footage of the video signals processed by FPGA and ASIC, then immediately hit the screen. This path delay minimum, but the most difficult. Write FPGA is not difficult, is courage.
This path is ideal for developing AR applications.
As shown in the figure, head of FPGA based on needs on the streaming video to connect on the PC, and then, through FPGA digital model of information fusion attitude to the original high-definition picture. In this way, we can greatly reduce the delay of real scenes, but also most of the computing resources that need to save, thus reducing your computer’s configuration requirements.
Attentive readers have discovered that this path is the path of VMG-PROV. This path is the most powerful, but extended to the highest. Video streaming through FPGA/ASIC and then into the computer, and finally after the screen is displayed. Soft mode suitable for early verification and experiment, written out by Soft optimization of the project according to the application requirements, and in other modes to achieve. Fendi
Light patterns are well understood, is the VR head with Visual SLAM. Camera, though not in the front of the imaging, but the time to detect the Translational motion of the head, and test methods within the Lighthouse from the outside are not the same. We continued the development of better SLAM will also open source, I hope you wait a bit longer.
Wrote it, is about it. As a small team, we want to be your dream of geeks. I believe that sharing our many spirits, spirit, heart, ability, and to support our people. We would like to share their work with such people, this is the most important reason we’re going open source.