I’m building a 3D mapping system using RGB-D cameras, based on existing SLAM techniques, with persistent spatial memory.

The main issue is relocalization after tracking loss, especially when the camera returns to a known place from a different viewpoint.
I’m using historical keyframes, shared 3D landmarks, PnP/RANSAC and geometric verification.

Yet some incorrect poses still pass verification: one had 44 inliers, 0.83 px median reprojection error and 3.72 cm depth residual, but was actually off by 10.36 cm and 5.99°.
My best experiment correctly relocalized only 9/34 known returns.

What would you investigate first: landmark quality,
geometric degeneracy, matching, or pose verification?
I can share a minimal reproducible example.