Language / 言語: [日本語](link to be added after publishing) · English
Fullseye in Motion: Screen Recordings, Physics Simulation and Inspection Numbers on Video
Many things in this library are quicker to see than to read. The YouTube channel holds short vertical videos, and this article is the guide to them. Each time a video is published, it is added to the matching section. The short videos are gathered in the channel's "Shorts" tab.
The description of each video says whether what you see is a simulation or real data. This article makes the same distinction in each section.
Four terms first
| Term | Meaning |
|---|---|
| Fullseye | An open-source Python library of image processing plus optics and physics simulation. Install with pip install fullseye
|
| Studio | Fullseye's point-and-click tool. Search for operators, stack them and try them while watching the result |
| Synthetic data | Images or signals made by calculation. Not real photographs, but the right answer is known, so they are good for measuring where a method fails |
| Reinforcement learning | A way for a robot to learn how to move by trial and error. The walking in the videos is the result of what was learned, not a replayed motion |
In plain words
- The videos fall into these groups: robots and muscle-driven bodies, a sports day, the Studio screen, optics and measurement, real data and 3-D, and checks of inspection numbers.
- Robot and muscle videos are physics simulations. They are not a real machine or a real human body.
- Inspection-number videos plant a known answer in synthetic data and show how far a method drifts from it.
Robots and moving bodies
This section holds: a humanoid that learned to walk with reinforcement learning, walking past obstacles, a bone-and-muscle model walking, a failed attempt in which it cannot yet stand, a whole-body model with about 700 muscles dancing, running, swimming and jumping, and a robot arm taking parts out of a bin.
What you see is a physics simulation, not a real machine or a real human body. Attempts that did not work are kept and shown as records.
Walking
A physics simulation of a humanoid that learned to walk with reinforcement learning.
Also a physics simulation: it walks past obstacles.
More videos:
Bodies driven by muscles
A physics simulation of a whole-body model with about 700 muscles, dancing.
The same whole-body model running.
More videos:
- Swimming with 700 muscles
- Jumping with 700 muscles
- Seeing the activity of 700 muscles as a skeleton walks
Robot arm
A physics simulation of a robot arm taking parts out of a bin.
It scores grasp candidates and picks one; only parts that left the bin count as success. A simulation on synthetic data.
Sports day
A humanoid walking 100 m without falling, a 17 cm staircase it takes on, jumping, dancing, a four-legged robot walking and sumo, an arm picking something up, and a blooper reel of the scenes that did not work.
Most are physics simulations. The dance is the exception: it replays a recorded human motion, with no physics calculation. The footwork in the sumo is a fixed motion, not learned. Each video description says which is which.
A simulation of a humanoid walking 100 m.
It takes on a 17 cm staircase.
More videos:
- Robot dance
- Robot jump
- Robot sumo
- Robot blooper reel
- A four-legged robot walks
- A robot arm picks it up
Using Fullseye Studio
Recordings of the real Studio screen: opening an image, searching about 1,000 operators by name, inserting one and following the result with a slider; stacking several steps and checking each step's result; and a visually built pipeline turning into Python code.
The screen is not a mock-up; it is the actual Studio, recorded as it is.
Open an image, search an operator and follow the result with a slider. This is a recording of the real Studio screen.
Build a 4-stage pipeline, inspect each stage, then remove and reorder.
More videos:
- From image to result in a few clicks
- Build an image pipeline visually
- Fullseye Studio: a visual pipeline becomes Python code you can hand to your IDE
- Your clicks become Python code
- Fullseye Studio: search ~1,000 operators, read the help, insert the result
- Find one of ~1,000 operators fast
- Fullseye Studio: orbit the asteroid Itokawa and a CT volume in the 3-D viewer
Optics and measurement
The principle of phase-measuring deflectometry that finds a dent by reflecting stripes in a glossy surface, the Talbot effect in which the image of a fine grating reappears again and again as light travels, and the concentric rings in an electron-microscope image dancing as the focus changes. All three are here.
All of these are synthetic images drawn by calculation, not real measurement data. They are numerical evaluations of wave-optics and phase formulas.
Stripes reflected in a glossy surface reveal a dent that a photo cannot show. A synthetic video drawn by calculation.
The image of a fine grating reappears again and again as light travels (the Talbot effect). Synthetic.
More videos:
The related article on non-coaxial optics will be linked here once its public URL is known.
Real data and 3-D
Videos made from real survey and wiring data, and videos that turn a solid built from formulas. For those using real data, the description names the data.
Ground change between 2012 and 2017, from real airborne laser surveys (LiDAR) of the same place, shown in colour.
A stimulus spreading over the measured wiring of a fly brain, computed with a simple model. The wiring is real; the spreading is a simulation.
More videos:
- A full orbit of asteroid Itokawa, rendered physically
- Only edges light up: an event camera simulation
- Spin the object while the hue goes from 0 to 360 degrees
- A procedurally built shape in metal, one full turn
- A hand skeleton of 27 bones, one full turn
- A still life made only from formulas, one full turn
Inspection numbers
A method looking right and a method giving right numbers are different things. The videos in this section plant the right answer in synthetic data and show how far an inspection or measurement method drifts from it. All are simulations on synthetic data.
Twelve transport robots jam even with an optimal plan
A plan for 12 vehicles was executed 1000 times, with every vehicle delayed with probability 0.3 at each step. Naive execution ("move if the next cell is free") jammed in 641 of 1000 runs; execution that follows the action dependency graph had 0 collisions and 0 deadlocks.
Alignment can create a dent that is not there
After a rigid alignment of a scanned point cloud to CAD, the centre of a part whose true deviation is only 1.2 µm showed a 121 µm dent. The alignment absorbs the mean of the bow, so the whole part reads as shifted down. Even a defect-free control showed 66.5 mm² of false out-of-tolerance area.
The same reprojection error, a focal length 281 times worse
With the calibration board tilted 32 degrees or 2 degrees, the reprojection error falls to 0.069 px in both, yet the error of the focal length fx is 0.026 % versus 7.33 % (281x). For an almost frontal setup, the reprojection error says nothing about the focal length.
How these are made
-
Tool: made with Fullseye (
pip install fullseye), which contains image processing, optics calculation, physics simulation and measurement parts. - Studio videos: the real Studio screen, captured offscreen. No mock-ups.
- Robots and muscles: physics-simulation footage. Not a real machine or a real human body.
- Synthetic data: a video that says "synthetic data" shows images or signals made by calculation, not real photographs.
- Videos using real data: for videos that use published real data (real surveys, a fly brain's wiring and so on), the description names the data and says so. Licences follow the terms of each data source.
The data named in the video descriptions:
- Fly brain wiring: MaleCNS v1.0 (Janelia FlyEM), CC BY 4.0. Computed on a subgraph of the top 3,000 neurons and overlaid on all cell-body positions (modified).
- Ground change: U.S. Geological Survey 3D Elevation Program (3DEP) LiDAR, public domain. The 2012 and 2017 height difference is shown as colour (modified).
- Asteroid Itokawa: a real shape model (public domain) with synthetic surface relief and boulders added.
Fullseye is licensed Apache-2.0. The source is in the GitHub repository.