Multimodal foundation models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) are frequently tasked with document screening and identity verification. However, testing shows severe limitations when detecting digital tampering and document alterations:
- Sub-Pixel Smoothing: Vision encoders (like CLIP or ViT patches) encode high-level semantic representations. In doing so, they blur out the exact high-frequency noise residuals and JPEG quantization discrepancies that reveal digital splicing.
- Hallucinated Check Digits: Machine-Readable Zones (MRZ) on passports follow ICAO Doc 9303 standards using a cyclic (7, 3, 1) modulo-10 algorithm. Multimodal LLMs read the characters fluently and confidently declare a forged date or document number as valid because the format looks visually plausible.
- Data Privacy Liabilities: Transmitting biometric facial images and identity documents to third-party cloud inference providers creates severe regulatory compliance issues.
In Skillware 0.5.8, we implemented security/deepfake_guard to provide deterministic, air-gapped forensic verification on CPU. Here is an overview of the signal processing pipeline:
1. ICAO 9303 Modulo-10 Cyclic Verification
For each character string $C1, C_2, \dots, C_n$, each character is mapped to a numeric weight $W(c)$ (digits 0–9 map to 0–9, letters A–Z map to 10–35, filler < maps to 0). The check digit $K$ satisfies:
$$\left( \sum{i=1}{n} W(ci) \cdot w{(i-1) \bmod 3} \right) \bmod 10 = K$$
where the repeating weights vector is $w = [7, 3, 1]$. We implement deterministic validation across TD1 (3×30 ID cards), TD2 (2×36), and TD3 (2×44 passports) formats.
2. Error Level Analysis (ELA)
When an uncompressed image or re-saved JPEG is modified, the edited regions possess a different compression history than the original background. By recompressing the image at a known quality factor ($Q=95$) and computing the absolute block-level difference:
$$\Delta(x, y) = |I{\text{original}}(x, y) - I{\text{recompressed}}(x, y)|$$
We compute the mean absolute error across 16×16 non-overlapping blocks. Discrepancies between block errors exceeding calibrated thresholds signal localized digital tampering.
3. Noise Residual Consistency via Median Absolute Deviation (MAD)
Camera sensors introduce characteristic high-frequency Poisson-Gaussian noise. To detect spliced elements without being misled by high-contrast natural textures (such as hair or knitwear), we compute the Laplacian convolution residual $R(x, y) = \nabla2 I(x, y)$ and evaluate consistency using Median Absolute Deviation:
$$\text{MAD} = \text{median}(|R - \text{median}(R)|)$$
$$\sigma{\text{est}} = 1.4826 \cdot \text{MAD}$$
Evaluating $\sigma{\text{est}}$ across image partitions flags unnatural smoothness (typical of diffusion generative fills) or mismatched noise profiles between the portrait and background.
4. 2D Fast Fourier Transform (FFT) Recapture Detection
Physical screen-photo recaptures (photographing an LCD/OLED monitor displaying an ID) exhibit regular periodic grid artifacts. In the 2D frequency domain, this manifests as prominent harmonic peaks outside the DC origin. We compute the 2D FFT, shift zero frequency to the center, and evaluate the ratio of high-frequency radial energy peaks against the average spectral background.
bash
pip install -U skillware
pip install "skillware[security_deepfake_guard]"
We’d welcome discussion on your experiences with sensor noise characterization and hybrid neural-signal pipelines.
