Using Screaming Frog’s Custom Extractor To Crawl The Website For Embedded Forms

Black and white photo of Ron Geronimo Posted by: Ronel Geronimo Date Posted: December 3, 2025
Back to Case Studies

Issue

Forms being missed during the mapping process and, in turn, being auto-detected as ‘General’ leads

Solution

During the setup process, we, MarTech Analysts, rely on the forms being listed to map. However, there are times when forms are missed during the mapping process, and when they are filled out, they will be tracked as ‘General’ leads in LeadmanagerFX.

To help avoid this, we can use Screaming Frog. Screaming Frog is a website crawler that SEO specialists use to analyze a site’s structure, find technical issues like broken links, and assess on-page elements like title tags and meta descriptions.

Screaming Frog is free to download. The license can be found in Keeper.

To crawl the website for embedded forms, we can use Screaming Frog’s Custom Extraction.

However, before you crawl a website, you need to configure the custom extraction first. To configure Custom Extraction, go to Configuration > Custom > Custom Extraction, then add these:

Extractor Name Mode of Extraction Syntax Select the Data to Extract
Form Count XPath count(//form) Function Value
Form Element CSSPath form Extract HTML Element

 

Screenshot link: https://webpagefx.mangoapps.com/mlink/file/MjM0Nzg3NDg=

Once added, feel free to save this configuration so you don’t have to enter these every time you crawl a website.

You can then start crawling the website. For the results, go to the Custom Extraction tab. The Form Count 1 column will show how many forms were detected in a page, while the Form Elements tab will show information about each form. The Address column shows the URL where the form is detected.

Screenshot link: https://webpagefx.mangoapps.com/mlink/file/MjM0Nzg3NDc=

It’s important to note, though, that since this custom extraction looks for the form attribute during the crawl, it would also find hidden forms, so make sure to check the page if an actual form appears.

Also, this configuration would not work on iframe forms. I am currently testing some XPaths for iframe forms, but because iframes are implemented differently on every website, I have not found a solution that works for all of them.

 

 

Achievement

Ensuring all embedded forms are mapped, which, in turn, would reduce General leads from auto-detected forms.

Snippet/Screenshot/Link