Learning Open-World Visual–Tactile Grasp Stability Prediction with Synthetic Data

Beining Han*, Gan Luyang*, Derek Geng, Abhishek Joshi, Jia Deng

Manuscript · 2025

A model trained on a large real-world visual-tactile dataset predicts grasp success on unfamiliar objects at 53%, which is a coin flip. Trained on data we generated in simulation, the same model reaches 77.5%. Real data is supposed to be the gold standard, and the reason it loses here is not that there was too little of it.

This paper was never published, so this page is the full account of it.

Background, and the questions

Whether a grip holds is settled by friction, contact area, and how force distributes across the fingers. A camera sees none of these, which is why grasp stability is normally predicted with a tactile sensor: a GelSight-style soft gel behind a camera, imaging its own deformation as it presses against the object.

Those sensors are also what makes the data hard to come by. Visual-tactile datasets are collected by hand, and each one ends up bound to a single gripper, a fixed camera position, one sensor model and a modest set of objects. A predictor trained on such a dataset inherits every one of those constraints. For the open-world case, where the object and the room are both unfamiliar, no feasible amount of further collection undoes that.

Generating the data instead raises three questions.

1. Can a simulator be faithful enough that its grasp outcomes match reality? Contact mechanics is the hardest thing for a simulator to get right, and it is precisely what the label depends on.
2. Does training on synthetic data beat training on real data once the test set is open-world?
3. Is touch earning its place, or would vision alone have done as well?

The simulator

Prior synthetic tactile data comes from rigid-body engines. We use FEM instead, on Taccel, which is built on the IPC contact framework, because the quantity we need to get right is the deformation of a soft gel against a rigid object.

The gripper is reduced to what matters: two tetrahedral FEM bodies, 2.5 × 2 × 0.4 cm, standing in for the gels of two GelSight Mini sensors, with Young’s modulus 10⁶ Pa and Poisson ratio 0.3. Vertices on the back of each gel are active nodes driven by a target velocity at kinematic stiffness 5 × 10⁴; everything else is passive and free to deform. The object is loaded as a rigid ABD body carrying its own mass and friction coefficient.

A grasp is a process, not an instant

The part that took the most iteration is that a grasp cannot be simulated as a single closing motion.

1. Finger closure. Gels close at 1 cm/s. At each step we sum the contact force over all passive vertices along the closing direction. When both gels exceed a stopping threshold, or the fingers fully close, the stage ends.
2. Force adjustment. Early gel-object contact produces wildly unstable forces, visible as the spike in the plot above. So the gripper spends 1.5 s in closed loop, opening at 0.2 cm/s when the mean force runs above threshold and closing at 0.2 cm/s when it runs below, until the force settles.
3. Lifting. The gripper rises at 2 cm/s to 5 cm, still regulating force. The grasp counts as a success if every point of the object ends more than 2 cm off the ground.

The training sample is rendered from the last frame of the force adjustment stage, which is the moment a real predictor would have to commit: contact has stabilised, the lift has not yet happened.

Accuracy here is bought with time. A 0.01 s step, contact stiffness 3 × 10⁶ kg·s⁻², contact distance threshold 5 × 10⁻⁴ m, and a CG solver held to 5 × 10⁻⁵ tolerance over up to 10⁴ steps make the simulation slow, but they are what keep the contact force consistent enough to be worth labelling.

The tactile renderer

Taccel’s built-in renderer casts rays at the tetrahedron surface to build a depth map. With gels meshed as coarsely as ours, that surface is faceted, so the depth map inherits a checkered pattern and a contact patch larger than the real one.

The fix is small and worth stating plainly: cast the rays at the object mesh instead. We know the object’s geometry, its pose, and the pose of the image plane, so the intersection can be computed directly and clipped by gel thickness, with RGB rendered in OpenGL over LED colours and positions tuned against real readings.

Objects, textures and light

Meshes and grasp poses come from ACRONYM, which supplies neither texture nor physical properties. We prompt GPT-4o for a plausible texture type from the object category, generate the UV map with Paint3D, and prompt GPT-4o again for mass and friction given category, dimensions and texture.

Visual frames are ray-traced in IsaacSim Replicator with the full gripper geometry reconstructed from the gel poses, randomised ground textures, and HDRI dome lighting from PolyHaven at random intensity. The camera sits eye-in-hand, matching the real rig, with pose and focal length jitter.

What came out

453 objects, 10k+ collision-free grasps simulated at two stopping thresholds (10 N and 40 N), three randomised renders each. 30k+ visual-tactile pairs from 10k+ unique grasps, costing roughly 2,500 GPU hours of physics and another 20 for rendering. The raw result skews positive and skews toward easy categories such as mugs, so the training set is resampled to balance both.

The real data

No open-world visual-tactile grasping dataset existed, so we built the rig and collected one. Its mechanics are based on the UMI gripper, with the fingers modified to carry the tactile sensors.

Because a person actuates it, data can be collected anywhere, at whatever grip force a hand happens to apply, which is the point.

Open-world set. 60 everyday objects, some from YCB, deliberately including transparent and reflective ones such as glasses and forks, across 10 real scenes: classrooms, kitchens, offices. The operator picks a grasp, squeezes, saves the frame before lifting, then lifts and records the outcome. 333 grasps, 163 failures and 170 successes, split 67 for validation and 266 for test.

Grasp-annotated set. Five 3D-printed objects with deliberately awkward geometry, some from EGAD, the kind usually called adversarial for grasping. 20 grasps each, with ground-truth grasp poses recovered from ArUco markers on a board and on the gripper. This set exists to compare simulators, which needs the grasp pose known exactly.

1. Fidelity

Replay all 20 grasps per object in both simulators and count how often the simulated outcome matches what really happened. Neither friction nor grip force is known, so both simulators get the same treatment: a grid search over friction coefficient and stopping force, with the best combination reported.

The failures cluster on B1, C4 and ORG, the three most awkward shapes, and there are two reasons.

Rigid-body engines need convex decomposition for collision detection, so an object with concavities becomes a union of convex pieces, and the approximation error lands on the contact surface, the one place the outcome is decided. Second, a real gel wraps around what it touches, spreading contact over a surface; rigid-rigid contact between a flat fingertip and an object gives you a point instead. FEM avoids the decomposition entirely and models the wrap directly.

Fidelity is worth having on its own terms, but it only matters here if it survives into a trained model.

2 and 3. Transfer, and whether touch earns its place

The predictor is deliberately ordinary, so that the data is what is being compared. Tactile input is the change in reading, grasp minus initialisation, rather than the raw image. Two 240 × 320 tactile images and one 240 × 320 RGB frame each go through a 4-layer CNN with BatchNorm and ReLU into 512-dim embeddings; the three are concatenated and read out by a 2-layer MLP with a sigmoid. Adam at 3 × 10⁻⁴, five seeds, checkpoint chosen on validation accuracy, evaluated on the held-out real test set.

Two baselines. Real is the Calandra et al. dataset: 9.2k grasps over 106 objects, collected on a Sawyer mount with a different GelSight model and a third-person camera. Taxim is a synthetic set built to be as close to ours as possible, same objects, physical properties, textures, backgrounds, grasp poses and balancing, differing only in the simulator and tactile renderer, at 33k+ pairs from 11k+ grasps.

Real data lands at chance, and not because it is small or careless. It is bound to one embodiment, one sensor, one camera angle, one object set, and open-world test objects fall outside all of them. The distribution shift, not the sample count, is what costs the accuracy.

Taxim is synthetic too, and reaches 61%. Generating data is not what helps. Generating data whose contact physics is right is. The fidelity result and the transfer result are the same result seen twice.

Vision alone scores 53.2%, indistinguishable from guessing. The cause is mundane and instructive: the camera is eye-in-hand, so at the moment of grasp roughly half the object is occluded by the gripper. There is often not enough of the object left in frame to infer its geometry, let alone its mass or friction.

Touch alone reaches 61.1% but swings wildly between seeds. That is a model which sometimes works, not one that works. Only the pair reaches 77.5%, with the tightest spread of anything we ran. The reading is that tactile carries the contact and the force while vision carries category and texture, which are proxies for mass and friction, and the prediction needs both.

Takeaway

The bottleneck was never volume. It was whether the process producing the data gets contact right, and that is a property you can lose by choosing a faster simulator. Fidelity at the point of contact is what this synthetic pipeline was actually buying.

The honest limitation is that all of this is one gripper, two GelSight Minis, and a parallel-jaw geometry. The claim that synthetic data transfers better than real data is established here for that embodiment, on 333 real grasps. Whether the same holds for other hands and other sensors is exactly the kind of thing the pipeline was built to make cheap to test, and it has not been tested yet.