Skip to content

Scanning Engine v2 - Module - Web v2

Overview

Targeting

Configuration

Only the publicly-available configuration keys that can be set in a job's module invocations will be described below. Additional configuration keys may exist, but not be shown here because they are restricted to specific users or because they are permanently set as a static value. If no keys have (required) after their names then invocations of this module need not contain a config key.

Named Keys

addresses-only

Discard hostnames once they are resolved to addresses.

  • Type: boolean

blocklist-files (required)

File of addresses, hostnames, and CIDR blocks that are blocked.

  • Type: list(string)

body

Controls whether to fetch the body or not.

  • Type: boolean

body-inline

Controls whether to include the body content inline.

  • Type: boolean

check-fragility

If set, skip subsequent paths if a target is determined to be fragile.

  • Type: boolean

chrome

The path to the chrome binary executable.

  • Type: string

chromedriver

The path to the chromedriver binary executable.

  • Type: string

connect-timeout

Timeout for each connection, in seconds.

  • Type: integer
  • Default: 15

custom-http-headers

Define custom HTTP headers to use in request.

  • Type: string

data

Define custom payload to send with request.

  • Type: string

debug

Run module with a debugging configuration.

  • Type: boolean

favicon-hash

Controls whether to include the favicon hash inline.

  • Type: boolean

favicon-inline

Controls whether to include the favicon base64 digest inline.

  • Type: boolean

follow-meta

Follow HTML meta tag and JavaScript redirects.

  • Type: boolean

host-header

Define a custom host header to use in request.

  • Type: string

http-method

Define a custom HTTP method to use in request, instead of GET.

  • Type: string

http-paths

Custom HTTP paths to request.

  • Type: list(string)
  • Default: ['/']

https

Use HTTPS instead of the regular HTTP.

  • Type: boolean

internal

If set, uses internal s3 buckets to store screenshots.

  • Type: boolean

max-v4

Maximum number of resolved IPv4 addresses to use.

  • Type: integer

max-v6

Maximum number of resolved IPv6 addresses to use.

  • Type: integer

render

Render web content and grab screenshot.

  • Type: boolean

request-timeout

Maximum timeout for each DNS request. (e.g., '10m', '1h', '1s', '1h10m')

  • Type: string

resolution-timeout

Maximum timeout for DNS resolution of each target. (e.g., '10m', '1h', '1s', '1h10m')

  • Type: string

resolver-address

Custom DNS resolver address to use.

  • Type: string

resolver-concurrency

Number of concurrent goroutines to use when resolving targets.

  • Type: integer

single-address

Use a single address from the DNS resolution.

  • Type: boolean

targeting-timeout

Maximum duration allowed for parsing and resolving all targets. (e.g., '10m', '1h', '1s', '1h10m')

  • Type: string

temporary-directory

Place the per-execution temporary directory path in the environment.

  • Type: string

user-agent

Define a custom user agent header to use in request.

  • Type: string

webapps

Analyzes web app information.

  • Type: boolean

worker-id

Place the ID of the Minion worker in the environment.

  • Type: integer

workers

Number of workers (threads) to run.

  • Type: integer
  • Default: 80

Schemas

The schema for the body object of all results generated with .task.module_name equal to webv2 can be found here. The schema for results is available both in standalone and bundled form.

Examples

These are examples of the .body object for results with .task.module_name equal to webv2.

HTTPS with BinaryEdge

This example was generated with the BinaryEdge website.

{
  "request": {
    "url": "https://binaryedge.io/",
    "path": "/",
    "method": "GET",
    "headers": {
      "accept-encoding": "gzip, deflate",
      "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36",
      "host": "binaryedge.io"
    }
  },
  "response": {
    "url": "https://binaryedge.io/",
    "path": "/",
    "protocol_version": "HTTP/1.1",
    "redirects": [],
    "status": {
      "code": 200,
      "message": "OK"
    },
    "headers": {
      "headers": {
        "date": "Fri, 06 Sep 2024 18:12:02 GMT",
        "content-type": "text/html",
        "transfer-encoding": "chunked",
        "connection": "keep-alive",
        "vary": "Accept-Encoding",
        "x-amz-id-2": "LH6Z6wun0e4TjK6HRk7RV8mTtOX9kl0nyV/Wcq1RnEPj7qkL3lY1JJ/iLmt1TQhdIPcNnsalrCw=",
        "x-amz-request-id": "THSZ27TWBXJ15BYF",
        "last-modified": "Mon, 24 Jun 2024 21:34:35 GMT",
        "access-control-allow-credentials": "true",
        "access-control-allow-methods": "GET, POST, OPTIONS, PUT, DELETE, PATCH",
        "access-control-allow-headers": "DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization,X-Token,External-ID,OTT,Coalition-App,x-datadog-trace-id,x-datadog-parent-id,x-datadog-origin,x-datadog-sampling-priority,x-datadog-sampled",
        "x-content-type-options": "nosniff",
        "permissions-policy": "microphone=(), geolocation=(), usb=(), payment=(), camera=()",
        "strict-transport-security": "max-age=31536000",
        "referrer-policy": "strict-origin-when-cross-origin",
        "content-security-policy": "default-src 'self' https: wss: data:; style-src 'self' 'unsafe-inline' blob: https://use.typekit.net https://p.typekit.net https://fonts.googleapis.com https://cdnjs.cloudflare.com https://cdn.jsdelivr.net; script-src 'self' 'unsafe-inline' 'unsafe-eval' blob: https://js.stripe.com https://www.google.com https://www.gstatic.com https://edge.fullstory.com https://maps.google.com https://maps.googleapis.com https://www.googletagmanager.com https://www.googleadservices.com https://www.google-analytics.com https://connect.facebook.net https://static.ads-twitter.com https://analytics.twitter.com https://googleads.g.doubleclick.net https://cdn.jsdelivr.net https://widget.intercom.io https://js.intercomcdn.com https://cdn.hellosign.com https://cdn.segment.com https://cdn.cookielaw.org https://snap.licdn.com https://ws.zoominfo.com https://grow.clearbitjs.com; img-src 'self' data: https://info.coalitioninc.com https://images.assets.sand.binaryedge.io https://be-riskportal-resolution-requests.s3.amazonaws.com https://s.gravatar.com https://mcusercontent.com https://d3f9qnon04ymh2.cloudfront.net https://d1ngxp4ef6grqi.cloudfront.net https://d3qx8u8yhl2krn.cloudfront.net https://be-resources.s3.eu-west-1.amazonaws.com https://images.ctfassets.net https://cdnjs.cloudflare.com https://static.intercomassets.com https://px4.ads.linkedin.com https://www.google.com https://maps.google.com https://maps.gstatic.com https://maps.googleapis.com https://www.google-analytics.com https://px.ads.linkedin.com https://www.facebook.com https://grow.clearbitjs.com https://t.co https://fastapi.tiangolo.com; font-src 'self' data: https://cdnjs.cloudflare.com https://fonts.gstatic.com https://js.intercomcdn.com https://secure.gravatar.com https://use.typekit.net; object-src 'self'; frame-src 'self' data: blob: https://www.google.com https://js.stripe.com https://app.hellosign.com https://player.vimeo.com https://coalitioninc.auth0.com https://*.binaryedge.io; frame-ancestors https://*.binaryedge.io https://*.coalitioninc.com https://*.thecoalition.com",
        "cf-cache-status": "DYNAMIC",
        "cross-origin-opener-policy": "same-origin",
        "server": "cloudflare",
        "cf-ray": "8bf06ccb3f5a8447-YVR",
        "content-encoding": "gzip"
      },
      "header_order": "date,content_type,transfer_encoding,connection,vary,x_amz_id_2,x_amz_request_id,last_modified,access_control_allow_credentials,access_control_allow_methods,access_control_allow_headers,x_content_type_options,permissions_policy,strict_transport_security,referrer_policy,content_security_policy,cf_cache_status,cross_origin_opener_policy,server,cf_ray,content_encoding",
      "header_order_md5_hash": "ee792487def15cb4eab327db1d46e9f5"
    },
    "body": {
      "sha256_hash": "86b0df437e9cbd9c85af2706500bd9a95ccf0bc0d7a1ca37f6ccf2a8f01f9202",
      "ssdeep_hash": "3338343a794f5546426264546d6976704367566a61527069302b6671656d6139665a4c45665570514d36474d6752656d346c467a5352734c4d3a52566976702f67386334665a4c45665570514d36474d67556d346c467a535254"
    },
    "title": "BinaryEdge"
  }
}

HTTPS with Coalition

This example was generated with the Coalition website.

{
  "request": {
    "url": "https://coalitioninc.com/",
    "path": "/",
    "method": "GET",
    "headers": {
      "accept-encoding": "gzip, deflate",
      "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36",
      "host": "coalitioninc.com"
    }
  },
  "response": {
    "url": "https://www.coalitioninc.com/",
    "path": "/",
    "protocol_version": "HTTP/1.1",
    "redirects": [
      {
        "redirect_uri": "https://coalitioninc.com/",
        "status": {
          "code": 308
        },
        "headers": {
          "Cache-Control": "public, max-age=0, must-revalidate",
          "Content-Type": "text/plain",
          "Date": "Fri, 06 Sep 2024 18:08:44 GMT",
          "Location": "https://www.coalitioninc.com/",
          "Refresh": "0;url=https://www.coalitioninc.com/",
          "Server": "Vercel",
          "Strict-Transport-Security": "max-age=63072000",
          "X-Vercel-Id": "cle1::8hs9p-1725646124604-2369d24a81a9",
          "Transfer-Encoding": "chunked"
        }
      }
    ],
    "status": {
      "code": 200,
      "message": "OK"
    },
    "headers": {
      "headers": {
        "age": "18",
        "cache-control": "public, max-age=0, must-revalidate",
        "content-encoding": "gzip",
        "content-security-policy": "default-src 'self' *.sentry.io *.zoominfo.com *.facebook.com *.ctfassets.net *.algolia.net *.contentful.com *.vercel-insights.com *.google.com *.google-analytics.com *.cookielaw.org *.intercom.io *.onetrust.io *.onetrust.com *.clickagy.com wss://nexus-websocket-a.intercom.io *.mktoresp.com cdn.linkedin.oribi.io *.linkedin.com qf23dtaqm7-2.algolianet.com qf23dtaqm7-3.algolianet.com qf23dtaqm7-1.algolianet.com ekr.zdassets.com coalition1659361680.zendesk.com wss://widget-mediator.zopim.com api.control.stg.binaryedge.io api.control.dev.binaryedge.io api.control.coalitioninc.com api.control.dev.binaryedge.io api.control.demo.binaryedge.io api.control.stg.binaryedge.io coalition.zendesk.com rum.browser-intake-datadoghq.com *.auryc.com *.chilipiper.com *.coalitioninc.com maps.googleapis.com blob:; style-src 'self' 'unsafe-inline' *.visualwebsiteoptimizer.com app.vwo.com s3.amazonaws.com *.sentry.io fonts.googleapis.com use.fontawesome.com unpkg.com info.coalitioninc.com web.coalitioninc.com heapanalytics.com; script-src 'self' 'unsafe-inline' 'unsafe-eval' *.visualwebsiteoptimizer.com app.vwo.com *.sentry.io *.googleoptimize.com *.heapanalytics.com heapanalytics.com *.google-analytics.com *.googletagmanager.com *.cookielaw.org *.googleadservices.com *.facebook.net *.zoominfo.com *.ads-twitter.com *.licdn.com *.clearbitjs.com *.intercom.io *.clickagy.com *.intercomcdn.com *.doubleclick.net *.google.com *.gstatic.com prism.app-us1.com munchkin.marketo.net info.coalitioninc.com web.coalitioninc.com *.mktoresp.com static.zdassets.com widget-mediator.zopim.com *.chilipiper.com maps.googleapis.com *.auryc.com *.apollo.io aplo-evnt.com; font-src 'self' *.sentry.io fonts.gstatic.com use.fontawesome.com data: *.intercomcdn.com heapanalytics.com *.auryc.com; frame-src 'self' app.vwo.com *.visualwebsiteoptimizer.com *.youtube.com *.clickagy.com *.doubleclick.net intercom-sheets.com *.google.com info.coalitioninc.com web.coalitioninc.com videos.ctfassets.net videos.contentful.com vimeo.com player.vimeo.com *.chilipiper.com; connect-src 'self' *.visualwebsiteoptimizer.com app.vwo.com *.auryc.com heapanalytics.com *.google-analytics.com *.vercel.app *.algolia.net qf23dtaqm7-2.algolianet.com qf23dtaqm7-3.algolianet.com qf23dtaqm7-1.algolianet.com *.cookielaw.org ekr.zdassets.com *.zoominfo.com cdn.linkedin.oribi.io *.linkedin.com *.mktoresp.com *.onetrust.io *.onetrust.com coalition.zendesk.com widget-mediator.zopim.com wss://widget-mediator.zopim.com api.control.coalitioninc.com api.control.dev.binaryedge.io api.control.demo.binaryedge.io api.control.stg.binaryedge.io *.chilipiper.com *.sentry.io maps.googleapis.com *.coalitioninc.com *.trayapp.io aplo-evnt.com cdn.rive.app */@rive-app/[email protected]/rive.wasm unpkg.com/@rive-app/[email protected]/rive.wasm; img-src 'self' *.visualwebsiteoptimizer.com chart.googleapis.com wingify-assets.s3.amazonaws.com app.vwo.com * data:; media-src * data:; object-src 'none'; worker-src 'self' blob:; frame-ancestors 'self' https://app.contentful.com",
        "content-type": "text/html; charset=utf-8",
        "cross-origin-opener-policy": "same-origin",
        "date": "Fri, 06 Sep 2024 18:08:26 GMT",
        "etag": "W/\"15ae4b90d3gb0nf\"",
        "permissions-policy": "camera=(), microphone=(), geolocation=()",
        "referrer-policy": "strict-origin-when-cross-origin",
        "server": "Vercel",
        "strict-transport-security": "max-age=63072000; includeSubDomains; preload",
        "x-content-type-options": "nosniff",
        "x-matched-path": "/en-us/[[...slug]]",
        "x-powered-by": "Next.js",
        "x-vercel-cache": "HIT",
        "x-vercel-id": "cle1::iad1::pfnbs-1725646124791-6acec68a7ba7",
        "transfer-encoding": "chunked"
      },
      "header_order": "age,cache_control,content_encoding,content_security_policy,content_type,cross_origin_opener_policy,date,etag,permissions_policy,referrer_policy,server,strict_transport_security,x_content_type_options,x_matched_path,x_powered_by,x_vercel_cache,x_vercel_id,transfer_encoding",
      "header_order_md5_hash": "f76292e0afdb52997b1a180b380c77a6"
    },
    "body": {
      "sha256_hash": "8254d514e16b4baeb3065de6da4129351a863fa35fcae09fd0d5e4c93d5b063c",
      "ssdeep_hash": "3736383a467047787144306147563273625567504f6b4c723769516f466664455435655948634848464241787962785a2b6e4662634255786569517845777855783261633a464d6c6a5575466e476664455646484c4370"
    },
    "title": "Cyber Insurance | Active Insurance & Cybersecurity | Coalition"
  }
}

Changelog

v1.0.0 (2025-07-08)

  • Initial release of versioning for each module. All modules are being tagged with version number 1.0.0. Going forward:
  • Major version should be changed when there are changes that impact consumers or clients of the modules.
  • Minor version should be changed when there are additions which enrich or enhance the module but shouldn't affect consumers or clients.
  • Patch version should be changed when there are bugfixes.