Build an AI Signature Verification and Fraud Detection System

Imagine you work in the fraud investigation team of a bank.

A customer disputes a large transaction.

The authorization document contains their signature. At first glance, the signature looks legitimate.

But “looks legitimate” is not good enough.

Was the signature actually written by the customer?

Was a genuine signature copied from another document?

Was the document edited after signing?

And if three different forensic techniques disagree, which result should you trust?

This is where signature verification becomes much more interesting than simple image classification.

In this project, we will explore a hybrid AI signature verification and fraud detection system that approaches the problem like a small digital forensic team.

Instead of asking one AI model for a yes-or-no answer, the system investigates a questioned signature from three directions:

What does a deep-learning model think?
Does the physical structure of the handwriting match?
Does the surrounding document show signs of manipulation?

The project combines YOLO signature detection, SAM-assisted segmentation, OpenCV preprocessing, a ResNet-18 Siamese network, structural image analysis, document forensics, evidence fusion, and LLM-generated reports.

That combination is what makes this project worth studying.

What You’ll Build

Refer the GitHub link for full code : Hybrid-Signature-Verification-Fraud-Detection-Engine

By the end of this tutorial, you’ll understand how to build a pipeline that can:

✔ Find signatures inside documents automatically
✔ Extract and clean signatures before analysis
✔ Represent handwriting using deep-learning embeddings
✔ Compare signatures using a Siamese neural network
✔ Analyze handwriting structure independently of deep learning
✔ Search for signs of copy-paste or document manipulation
✔ Combine several imperfect signals into one forensic assessment
✔ Generate technical and executive-friendly investigation reports

Why Signature Verification Is Harder Than It Looks

Humans do not produce identical signatures every time.

Sign your name five times and compare the results.

The overall style will remain recognizable, but tiny details will move:

stroke length,
spacing,
pressure,
slant,
intersections,
loops,
proportions.

That creates a difficult AI problem.

A genuine signature can look different from another genuine signature.

Meanwhile, a skilled forgery can look surprisingly similar.

There is another problem.

Sometimes the handwriting itself is genuine.

The fraud happens because someone copied a genuine signature from one document and pasted it into another.

A pure signature-matching model might say:

These signatures are extremely similar.

Technically, it could be correct.

Forensically, the document could still be fraudulent.

That distinction explains the most important architectural decision in this project:

Signature verification and document fraud detection are related problems, but they are not the same problem.

The Big Idea: Don’t Trust One Model

Think about a hospital.

A doctor rarely diagnoses a difficult condition using one measurement.

They might combine:

blood tests,
imaging,
symptoms,
patient history.

Each measurement sees a different part of the problem.

This signature system follows the same philosophy.

One branch learns visual identity.

Another examines geometric structure.

A third investigates the document itself.

Only after all three have produced evidence does the system combine their conclusions.

That technique is often called evidence fusion.

It is one of the most useful ideas in real-world AI engineering.

Architecture

Here is the pipeline at a high level:

                Questioned Document
                        │
                        ▼
               Document Preprocessing
                        │
                        ▼
                Signature Detection
                     (YOLO)
                        │
                        ▼
              Signature Segmentation
                 (SAM when needed)
                        │
                        ▼
                Signature Cleaning
                    (OpenCV)
                        │
          ┌─────────────┼─────────────┐
          │             │             │
          ▼             ▼             ▼
   Learned AI      Structural AI    Document
     Branch            Branch       Forensics
          │             │             │
   ResNet-18         Skeleton       ORB Matching
   Siamese Net       Contours       Copy/Paste
   Embeddings        Hu Moments     ELA
   Cosine Sim.       Fourier        Noise
          │             │             │
          └─────────────┼─────────────┘
                        ▼
              Reliability-Aware
                Evidence Fusion
                        │
                        ▼
               Forensic Assessment
                        │
                        ▼
                  LLM Reporting
                 ┌──────┴──────┐
                 ▼             ▼
             Technical      Executive
              Report         Report

Let’s understand why every stage exists.

Step 1: Find the Signature Before Verifying It

Suppose you upload a scanned contract containing:

paragraphs,
tables,
stamps,
logos,
dates,
signatures.

You cannot simply feed the entire page into a signature verification model.

First, the system needs to answer:

Where is the signature?

This is an object detection problem.

The project uses YOLO for that stage.

Understanding Object Detection

Image classification asks:

What is in this image?

Object detection asks:

What is in this image, and where is it?

For a signature system, the desired output might conceptually resemble:

signature
confidence: 0.94
bounding box: [x1, y1, x2, y2]

The bounding box lets the pipeline crop the signature region and send only the relevant pixels downstream.

The repository points to Ultralytics’ Signature Detection Dataset for training. That dataset currently contains 178 annotated document images: 143 training images and 35 validation images, with a single signature object class. (Ultralytics Docs)

Why this stage matters

Poor detection poisons everything downstream.

If the crop includes:

nearby text,
stamps,
table borders,
or only half of the signature,

the verifier may compare document noise instead of handwriting.

That makes signature detection more than preprocessing.

It is the front door of the entire forensic system.

Step 2: Separate the Ink from the Document

Even a good bounding box can contain unwanted information.

Imagine this crop:

Approved by:   John Smith
               ~~~~~~~~~~
                  signature

You want the handwriting.

You do not want “Approved by,” the underline, a stamp, or background texture.

This is where segmentation becomes useful.

The repository uses SAM-assisted segmentation for difficult or “dirty” signatures.

Understanding Segmentation

Detection gives you a rectangle.

Segmentation tries to identify the actual pixels belonging to an object.

Think of the difference like this:

Detection
┌────────────────────┐
│    signature       │
└────────────────────┘

Segmentation
        ███
     ██    ███
   ██         ██

The Segment Anything Model, or SAM, was designed as a promptable image segmentation model capable of generating object masks across many types of imagery. (arXiv)

For signature analysis, segmentation can help isolate ink when the surrounding document is messy.

Step 3: Clean the Signature

Now we have the signature region.

But scanned documents are rarely perfect.

You may still have:

gray backgrounds,
compression artifacts,
scanning noise,
faint ink,
unwanted lines.

The project therefore uses OpenCV-based signature cleaning before comparison. (github.com)

Conceptually, preprocessing might turn this:

gray paper + text + faint signature + noise

into something closer to:

clean foreground signature
on a normalized background

This matters because every later algorithm depends on the quality of the extracted signature.

One useful principle appears here:

In applied AI, better input often beats a more complicated model.

Step 4: Compare Signatures with a Siamese Neural Network

Now we reach the first major AI branch.

The repository uses a ResNet-18 Siamese network for learned signature verification.

The repository recommends the HSig260 dataset for training this branch.

But what exactly is a Siamese network?

Understanding Siamese Networks

Imagine two identical detectives.

Each detective receives one signature.

They follow exactly the same inspection procedure.

At the end, each produces a compact description of what they saw.

We then compare those descriptions.

That is roughly how a Siamese network works.

Signature A ──► Neural Network ──► Embedding A
                     ▲
                     │ shared weights
                     ▼
Signature B ──► Neural Network ──► Embedding B

Embedding A
     │
     ├── similarity comparison
     │
Embedding B

The word Siamese refers to the fact that both branches share the same model parameters.

We are not asking the network:

Is this John’s signature?

Instead, we are asking:

How similar are these two signatures?

That difference is extremely useful.

Understanding Embeddings

An embedding is a numerical representation of an object.

Think of it as an AI-generated fingerprint.

Instead of storing the idea of a signature as millions of pixels, the neural network converts the handwriting into a vector such as:

[0.24, -0.71, 0.13, 0.82, ...]

Those numbers are not directly meaningful to humans.

But their position in embedding space is meaningful to the model.

Signatures that appear similar should ideally land near one another.

Different signatures should land farther apart.

The repository uses ResNet-18 as the neural architecture behind this process. ResNet-18 is part of the residual-network family available through TorchVision. (PyTorch Documentation)

Understanding Cosine Similarity

Once two embeddings exist, we need to compare them.

The project uses cosine similarity.

Imagine two arrows.

If they point almost exactly in the same direction, they are highly similar.

If they point in different directions, their similarity falls.

High similarity

     ↗ A
    ↗ B


Lower similarity

     ↑ A

        → B

Cosine similarity performs roughly that comparison in a space with many dimensions.

This gives the learned branch a quantitative signal describing how similar the questioned signature is to its references.

Why Use Several Reference Signatures?

The repository’s case runner asks for:

one questioned signature,
three reference signatures.

That is a smart design.

One genuine reference may contain an unusual variation.

Perhaps the person:

signed quickly,
used a different pen,
changed the size,
shortened their surname.

With multiple references, you begin capturing the natural variation of the person’s handwriting.

Conceptually:

Reference 1 ─┐
Reference 2 ─┼──► Genuine variation
Reference 3 ─┘

Questioned ──────► Compare against variation

This is much closer to a real verification problem than comparing two perfectly controlled images.

Step 5: Ask a Completely Different Question

Deep learning is powerful.

But forensic systems benefit from independent evidence.

So the project’s second branch deliberately looks at signatures through a different lens.

Instead of asking:

What visual representation has the neural network learned?

it asks:

Does the physical structure of the handwriting match?

The repository’s structural branch includes:

skeleton extraction,
Shape Context,
Fourier descriptors,
Hu Moments,
contour analysis,
graph-based comparison,
structural similarity.

OpenCV provides shape-analysis functionality including Hu moments and shape-context-related operations. (OpenCV Documentation)

Understanding Skeletonization

Imagine drawing a signature with a thick marker.

The stroke might be 15 pixels wide.

For structural analysis, thickness may be less interesting than the path itself.

Skeletonization attempts to reduce strokes toward their center lines.

Before:

████████
   ███████
       █████

After:

───────
   ───────
       ─────

Now algorithms can reason about:

branches,
junctions,
curvature,
endpoints,
connectivity.

In other words, the system begins analyzing handwriting as geometry.

Understanding Contours

A contour describes the boundary of a shape.

For signatures, contours provide information about:

loops,
curves,
proportions,
stroke regions,
overall form.

If two signatures look similar to a neural network but their geometric boundaries differ significantly, that disagreement becomes useful forensic evidence.

This highlights the value of hybrid AI.

Disagreement is not necessarily a failure.

Sometimes disagreement is exactly what you want the system to expose.

Understanding Hu Moments

Hu Moments are numerical descriptors of shape.

You can think of them as a compact geometric summary.

Instead of comparing every pixel, the algorithm creates numbers representing aspects of an object’s geometry.

The advantage is that these descriptors can remain useful even when shapes undergo some translation, scaling, or rotation.

For signatures, that means the system can compare underlying form instead of demanding pixel-perfect alignment.

Understanding Fourier Descriptors

A signature contour is essentially a complicated curve.

Fourier descriptors provide another way to encode that curve.

Imagine describing a complex coastline.

You could record every rock.

Or you could capture the larger pattern of bends and waves.

Fourier descriptors lean toward the second idea.

They provide a mathematical description of shape that complements the learned embedding from the Siamese network.

Step 6: Investigate the Document, Not Just the Handwriting

This may be the most interesting part of the project.

Suppose a fraudster steals a genuine signature from an older document.

They paste it onto a new contract.

The signature verifier receives:

Questioned signature = genuine copied signature
Reference signature  = genuine signature

What should happen?

The learned model may report extremely high similarity.

The structural branch may agree.

Yet the document is fraudulent.

This is why the repository contains a third branch dedicated to document forensics.

Its documented checks include:

copy-paste screening,
ORB feature matching,
template matching,
Error Level Analysis,
compression analysis,
noise consistency,
reference reuse detection.

Understanding Copy-Paste Detection

Digital copying can leave clues.

Suppose one signature region appears almost exactly somewhere else.

A feature-matching algorithm can search for strongly corresponding visual points.

The repository uses ORB feature matching as part of this investigation.

The important distinction is this:

Signature verification:

"Does this look like the same writer?"


Document forensics:

"How did this image get here?"

Those questions overlap, but they are not interchangeable.

Understanding Error Level Analysis

JPEG images are compressed.

When an image is edited and recompressed, different regions may accumulate different compression histories.

Error Level Analysis, commonly shortened to ELA, attempts to make such inconsistencies visible.

That does not mean:

ELA anomaly = definite fraud

Compression differences can occur for innocent reasons too.

Instead, ELA should be treated as another piece of evidence.

That philosophy matches this project’s broader approach: combine signals rather than turning a single heuristic into a verdict.

Understanding Noise Consistency

Scanners and cameras introduce tiny visual patterns.

When an image region comes from another source, its noise characteristics may differ from the surrounding document.

Imagine cutting a small square from one photograph and pasting it into another.

To your eyes, the images may blend.

At a lower level, their texture may not.

Noise analysis attempts to detect that mismatch.

Again, it is a clue rather than proof.

Step 7: Fuse the Evidence

We now have three investigators.

Investigator 1:
Deep-learning similarity

Investigator 2:
Structural similarity

Investigator 3:
Document manipulation evidence

What happens if they disagree?

This is where reliability-aware evidence fusion enters the pipeline. The repository explicitly lists reliability-aware fusion as a core feature and final output stage. (github.com)

Instead of blindly trusting one score, a fusion layer considers the available evidence together.

Conceptually:

Learned similarity ───────┐
                          │
Structural similarity ────┼──► Evidence Fusion ──► Assessment
                          │
Forensic anomalies ───────┘

The word reliability-aware is especially important.

Not every signal is equally trustworthy for every case.

For example:

Clean signature crop
→ learned similarity may be highly informative

Extremely noisy scan
→ learned similarity may deserve less confidence

Exact duplicate region detected
→ copy-paste evidence may deserve additional attention

This is a much better mental model for applied AI than:

model.predict() → truth

Step 8: Generate Reports with an LLM

At this stage, the analytical work is largely complete.

But there is another practical problem.

Different people need different explanations.

A machine-learning engineer may want:

similarity scores
structural metrics
forensic indicators
reliability information

A manager may want:

What happened?
How concerning is it?
What should we review?

The repository uses an OpenAI LLM to generate two outputs:

an executive stakeholder report,
a technical fusion report.

Notice where the LLM sits in the architecture.

It is after the forensic analysis.

That is an important design choice.

Evidence
   ↓
Fusion
   ↓
Structured Findings
   ↓
LLM
   ↓
Human-readable explanation

The language model becomes a communication layer rather than the primary signature verifier.

That separation is valuable for systems where traceability matters.

Putting the Complete Workflow Together

The repository is designed primarily for Google Colab.

Training and case analysis are separated into different notebooks. According to the project documentation, train_models_cells.ipynb trains the YOLO signature detector and ResNet-18 Siamese verification model. Once checkpoints exist, complete_signature_case_runner.ipynb runs an individual investigation.

A typical case looks like this:

1 questioned signature
          │
          │
3 reference signatures
          │
          ▼
     Preprocessing
          │
          ▼
   Signature Detection
          │
          ▼
   Cleaning/Segmentation
          │
          ▼
 ┌────────┼────────┐
 ▼        ▼        ▼
Deep   Structural Forensic
AI       AI       Analysis
 │        │        │
 └────────┼────────┘
          ▼
     Evidence Fusion
          │
          ▼
     Case Assessment
          │
          ▼
      LLM Reports

This is much closer to an investigation workflow than a conventional image-classification tutorial.

What Makes This Architecture Interesting?

At first, the project looks like a collection of computer-vision algorithms.

But there is a deeper lesson.

The architecture separates three different kinds of knowledge.

1. Learned knowledge

The Siamese model learns patterns from data.

It may notice subtle relationships that humans would struggle to define manually.

2. Engineered knowledge

Structural algorithms explicitly inspect shapes, contours, skeletons, and geometry.

Humans understand what these measurements are trying to capture.

3. Forensic knowledge

Document analysis looks beyond writer identity and investigates manipulation.

Those signals answer different questions.

Together, they create something stronger than simply stacking more neural-network layers.

Why Not Build One Giant Neural Network?

You certainly could experiment with an end-to-end model.

But there are tradeoffs.

Suppose the system marks a signature as suspicious.

A reviewer asks:

Why?

A giant model might provide one confidence score.

A hybrid system can potentially provide several interpretable observations:

Learned similarity: strong

Structural similarity: moderate

Copy-paste evidence: concerning

Compression consistency: abnormal

That does not automatically make the conclusion correct.

But it gives investigators a better starting point for review.

In sensitive domains, explainability is not merely about producing prettier charts.

It can help humans understand which evidence drove a decision.

Try It Yourself

Beginner Challenge

Run several genuine signatures from the same person through the verification pipeline.

Compare the similarity values.

Question to investigate:

How much does one person’s signature naturally vary?

Then try changing:

image brightness,
scale,
rotation,
background quality.

Observe which transformations hurt the system most.

Intermediate Challenge

Remove one branch at a time.

Run:

Deep learning only
Structural analysis only
Forensics only
Hybrid system

Compare their mistakes.

You are performing an ablation study.

An ablation study asks:

What happens when I remove one component?

This is one of the best ways to understand whether complicated architecture actually adds value.

Advanced Challenge

Build a calibration dataset containing:

genuine signatures
skilled forgeries
random forgeries
copied genuine signatures
edited documents
clean documents

Then evaluate each branch separately.

Measure more than accuracy.

Study:

false acceptance rate
false rejection rate
precision
recall
ROC curves
threshold sensitivity

Finally, calibrate the evidence-fusion strategy using real validation data instead of arbitrary weights.

That experiment would move the project significantly closer to production-grade evaluation.

Real-World Applications

Hybrid signature verification has potential applications wherever handwritten authorization still matters.

Banking

Banks can use signature verification to help prioritize questionable:

cheques,
withdrawal forms,
account instructions,
loan documentation.

The key phrase is help prioritize.

For high-impact decisions, automated verification should support qualified human reviewers rather than silently replacing them.

Insurance

Claims documents often contain signatures and scanned forms.

A forensic pipeline can highlight cases where both handwriting similarity and document-manipulation evidence deserve review.

Legal Document Processing

Contracts, affidavits, authorizations, and other signed documents can be screened for anomalies before manual inspection.

Government

Signed forms appear in licensing, benefits administration, identity workflows, and regulatory processes.

Enterprise Procurement

Organizations processing large numbers of purchase approvals or vendor documents can use similar techniques as part of fraud-risk screening.

What This System Can Do

Based on its documented architecture, this prototype can combine multiple forms of evidence around offline handwritten signatures, including deep-learning similarity, structural comparisons, and document-forensic screening. It can also generate technical and stakeholder-oriented reports from the resulting evidence. (github.com)

That makes it useful as:

an educational forensic-AI project,
a research prototype,
a suspicious-case prioritization system,
a foundation for more rigorous experimentation.

What It Cannot Prove

This distinction is critical.

A similarity score cannot prove authorship.

A structural match cannot prove authenticity.

An ELA anomaly cannot prove image manipulation.

And an LLM-generated explanation cannot turn uncertain evidence into certainty.

Real forensic handwriting examination involves context that an offline image pipeline may not possess.

For example, the system usually cannot directly observe:

pen pressure through time,
stroke velocity,
stroke order,
pauses,
writing dynamics.

Those signals require online signature capture, such as a digitizing tablet.

A scanned signature is an offline signature.

It only preserves the final image.

Important Ethical Considerations

A false positive could accuse an innocent person of fraud.

A false negative could allow fraud to pass unnoticed.

That makes threshold selection a business and risk decision, not merely a machine-learning optimization problem.

A responsible deployment should therefore include:

AI analysis
     ↓
Evidence presentation
     ↓
Human review
     ↓
Decision

rather than:

AI score
     ↓
Automatic accusation

Signature images are also biometric-like identity data and should be handled with appropriate access controls, retention policies, and privacy protections.

The Biggest Lesson: AI Systems Are Pipelines

Many beginner projects teach this pattern:

model.predict(image)

Real systems rarely end there.

This project shows a more useful mental model:

Acquire data
    ↓
Locate the relevant object
    ↓
Clean the input
    ↓
Create multiple forms of evidence
    ↓
Measure uncertainty
    ↓
Fuse evidence
    ↓
Explain the result
    ↓
Let humans review difficult cases

The neural network is only one component.

That is how many serious AI systems are designed.

Key Takeaways

If you remember only a few ideas from this project, remember these:

Signature verification is a similarity problem, not just a classification problem.

Siamese networks are useful because they learn representations that make pairs of signatures comparable.

Embeddings act like learned digital fingerprints.

They compress complicated images into vectors that can be compared mathematically.

Deep learning should not always work alone.

Structural algorithms can provide independent evidence about shape and geometry.

Authentic handwriting does not guarantee an authentic document.

A genuine signature can still be copied into a fraudulent document.

Evidence fusion is often more useful than chasing one magical model.

Different techniques see different failure modes.

LLMs can be valuable after prediction.

Their strongest role here is turning structured evidence into explanations for different audiences.

And perhaps most importantly:

AI should surface evidence—not manufacture certainty.

Next Project: Build a Signature Search Engine

Once signatures can be converted into embeddings, an interesting possibility appears.

Instead of comparing one questioned signature against only three references, imagine searching millions of stored signatures.

Questioned Signature
        ↓
     Embedding
        ↓
   Vector Database
        ↓
Nearest Signature Matches
        ↓
Forensic Review

That turns the project into a large-scale biometric similarity search system.

And it introduces the next major AI concept:

vector databases and approximate nearest-neighbor search.

Discover more from Debabrata Pruseth

Subscribe to get the latest posts sent to your email.