MQL is highly extensible and can integrate with virtually any tool or service to build better detection rules.

📘

Request a function!

Don't see a function you want? Let us know via email or Slack!

beta.message_screenshot

beta.message_screenshot() → File

The beta.message_screenshot function takes a screenshot of the message using the message body's HTML section. This screenshot is the same as the one that shows in the Message Preview pane when viewing a message, and is a representation of what the end-user would see. The resulting file can be passed into other File analysis functions, such as file.explode or ml.logo_detect:

// Check for an embedded Microsoft logo
any(ml.logo_detect(beta.message_screenshot()).brands, 
    .name == "Microsoft" and .confidence in ("medium", "high")
)

// Run OCR on a screenshot of the message
any(file.explode(beta.message_screenshot()),
    strings.ilike(.scan.ocr.raw, "*free cooler*")
)

View detection rules that use this function

File Analysis

Files can be delivered via email in a variety of ways, including directly as an attachment or auto-downloaded via links.

file.explode

file.explode(input: File | HTML) -> [FileExplodeOutput]

FileExplode uses Strelka, a file extraction and metadata collection system developed by Target.

Strelka uses a variety of scanners to parse files of a specific flavor and performs data collection and/or file extraction on them. Strelka can recursively extract nested files (like a Word doc within a Zip file), identify malicious scripts, suspicious executables and text, run analysis like OCR and Macro detection, and more. For more information on how Strelka works, see the official Strelka documentation.

For a list of all available scanners, see the Github repo or the official Strelka docs.

View detection rules that use this function

// detect HTML smuggling techniques
any(attachments, .file_extension in~ ('html', 'htm') and
  any(file.explode(.), "unescape" in .scan.javascript.identifiers) 
)

// detect encrypted zip files
any(attachments,
  any(file.explode(.), 
    'encrypted_zip' in .flavors.yara
  )
)

// detect attachments soliciting the user to enable macros using OCR
any(attachments,
  any(file.explode(.),
    strings.icontains(.scan.ocr.raw, "enable macros")
  )
)

// detect macros with auto-open
any(attachments,
  any(file.explode(.),
    any(.scan.vba.auto_exec, . == "AutoOpen")
  )
)

// detect macros calling an exe
any(attachments,
  any(file.explode(.),
    any(.scan.vba.hex, strings.ilike(., "*exe*"))
  )
)

file.html_screenshot

file.html_screenshot(input: File) -> File

The file.html_screenshot function takes a screenshot of HTML files so that you can query the image. This allows you to run logo detect on HTML attachments — ml.logo_detect(file.html_screenshot(.)) — or send the result to file.explode, empowering you to run OCR and QR analysis.

any(attachments,
    (
      .file_extension in~ ("html", "htm", "shtml", "dhtml")
      or .file_type == "html"
      or .content_type == "text/html"
    )
    and any(ml.logo_detect(file.html_screenshot(.)).brands,
            .name != null and .confidence in ("medium", "high")
    )
)

file.oletools

file.oletools(input: File) -> OleToolsOutput

Oletools, developed by Philippe Lagadec, analyzes Microsoft OLE2 files such as Microsoft Office documents for malware and other suspicious indicators.

Use file.oletools to analyze attachments for malware or suspicious indicators like VBA macros, remote OLE objects, encryption, and more.

View detection rules that use this function

// detect suspicious macros
any(attachments, file.oletools(.).indicators.vba_macros.exists)
any(attachments, file.oletools(.).indicators.vba_macros.risk == "high")

// detect potential attempts to exploit CVE-2021-40444  (https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-40444)
any(attachments, any(file.oletools(.).relationships, strings.ilike(.target, "*html:http*")))

// detect external OLE object relationships
any(attachments, file.oletools(.).indicators.external_relationships.count > 0)

// detect encrypted Office documents
any(attachments, file.oletools(.).indicators.encryption.exists)

// detect macros that attempt to auto-execute when the document is opened
any(attachments, any(file.oletools(.).macros.keywords, .type == "autoexec"))

// detect suspicious macro source code
any(attachments, strings.ilike(file.oletools(.).macros.vba_code_all_modules, "*kernel32*", "*GetProcessId*"))

file.parse_eml

file.parse_eml(input: Attachment) -> MessageDataModel

The file.parse_eml function takes in an EML attachment (file extension .eml or content type message/rfc822) and parses it into an MDM.

any(attachments,
    (.file_extension == "eml" or .content_type == "message/rfc822")
    and strings.icontains(file.parse_eml(.).subject.subject, "invoice")
)

file.parse_html

file.parse_html(input: File) -> HTML

The file.parse_html function parses an HTML file from an attachment, returning the full raw along with display_text, inner_text. This empowers detections such as running NLU on the display_text and completing a regex on the HTML without custom scanners or YARA signatures.

any(attachments,
    (
      .file_extension in~ ("html", "htm", "shtml", "dhtml")
      or .file_type == "html"
    )
    and regex.icontains(file.parse_html(.).raw,
                        "fromCharCode",
                        "charCodeAt",
                        "charAt",
                        "parseInt"
    )
)

Machine Learning functions

ml.link_analysis

ml.link_analysis(input: Link | URL, mode="default") → LinkAnalysisOutput

LinkAnalysis analyzes a link and classifies them as benign or suspicious. The service sends suspicious URLs to a headless browser which resolves the effective URL and collects a screenshot. The screenshot is sent to an object detection model to detect brand logos, buttons, and input forms. We chose Phishpedia, an Open Source object detection project as our baseline model architecture.

If any logos are detected, those logos are cropped from the original screenshot and compared to a set of protected brand logos commonly used in credential phishing attacks. Discovered brands are available to MQL, along with summary information about login input boxes or captchas in the screenshot.

mode is an optional argument that alters LinkAnalysis's analysis criteria (see note below). By changing mode from its default of "default" to "aggressive", LinkAnalysis performs extra processing on a link when determining whether to fully analyze the link. For example, LinkAnalysis with mode="aggressive" will fetch the destination link of known common click trackers via HEAD and apply normal analysis criteria to that destination link.

View rules that use this function

// detect links to credential phishing pages
any(body.links, 
    all([ml.link_analysis(.)],
        .credphish.disposition == "phishing"
         and .credphish.brand.confidence in ("medium", "high")
     )
)

// detect any links to credential phishing pages
any(body.links, 
    any([ml.link_analysis(., mode="aggressive")],
        .credphish.disposition == "phishing"
         and .credphish.brand.confidence in ("medium", "high")
     )
)

// detect free subdomain links with a login or captcha
any(body.links, 
    all([ml.link_analysis(.)], (
          .credphish.contains_login
          or .credphish.contains_captcha
     )
     and (
          .effective_url.domain.root_domain in $free_subdomain_hosts
          or .original_url.domain.root_domain in $free_subdomain_hosts
     ))
)

// analyze the final DOM of a link within the body
any(body.links, 
    strings.icontains(ml.link_analysis(.).final_dom.display_text, "Redirect Notice")
    and strings.contains(ml.link_analysis(.).final_dom.display_text, ".zip")
)

📘

Analysis criteria

In order to prevent LinkAnalysis from "clicking" on every link, such as Unsubscribes and one-time password resets, LinkAnalysis uses a URL classification model to determine which links to actually send to the service for analysis.

You can check whether LinkAnalysis submitted, retrieved, or analyzed the target page by inspecting the response in the MQL editor.

If you observe LinkAnalysis analyzing links it shouldn't or not analyzing links it should, please send us an email or post in the Slack Community.

ml.logo_detect

ml.logo_detect(input: File) -> [LogoDetectOutput]

LogoDetect uses computer vision to detect common brand logos used in attachment-based credential phishing attacks, such as impersonations of PayPal, Adobe, Microsoft, Outlook, Office365, DocuSign, and more. This includes embedded images in the body of messages as CIDs.

Our object detection model identifies logos, which are then cropped into separate images. These images are passed through a Siamese Neural Network to generate a feature vector. We compare this vector to a database of known logos using a similarity calculation. If the score exceeds a predetermined threshold, we confirm it as the respective brand logo.

For text-based logos, we utilize OCR, a computer vision technique for extracting text from images. Combined with Siamese Networks, this approach ensures comprehensive logo detection.

View Rules that use this function

// detect SharePoint logos in attached images
any(attachments,
    .file_type in ('png', 'jpeg', 'jpg', 'bmp')
    and any(beta.logo_detect(.).brands, .name == "Microsoft SharePoint")
)

// detect DocuSign logos in attached images
any(attachments,
    .file_type in ('png', 'jpeg', 'jpg', 'bmp')
    and any(beta.logo_detect(.).brands, .name == "DocuSign")
)

// detect Norton logos in attached PDFs
any(attachments,
    .file_type == "pdf"
    and any(beta.logo_detect(.).brands, .name == "Norton")
)

List of Supported Brands

ADP  
AT&T  
Adobe  
Amazon  
American Express  
Apple  
BB&T Corporation  
Bank of America  
Box  
Capital One Bank  
Captcha  
Chase  
ChicagoTitle  
Coinbase  
DHL  
Discover  
DocuSign  
Dropbox  
Ebay  
Facebook  
FidelityTitle  
FirstAm  
GeekSquad  
Generic Webmail  
Google  
GoogleDrive  
Gusto  
HSBC Bank  
Heroku  
Hulu  
IRS  
Instagram  
Key Bank  
LawyersTitle  
Ledger  
LinkedIn  
M & T Bank  
MadisonTitle  
Mastercard  
Meta  
Microsoft
Microsoft Office365  
Microsoft OneDrive
Microsoft Outlook
Microsoft SharePoint
Microsoft Teams
Navy Federal Credit Union  
Netflix  
Norton  
Okta  
OldRepublicTitle  
OVO
PayPal  
Quickbooks
Rakuten
SBB
Silicon Valley Bank  
Slack  
Spotify  
Square  
StewartTitle  
SunTrust Bank
Swiss Post
Swisscom
TD Bank  
TicorTitle  
U.S. Bank  
UPS  
Venmo  
Visa  
WeTransfer  
Wells Fargo  
WhatsApp  
Zoom

ml.macro_classifier

ml.macro_classifier(input: File) → MLMacrosOutput

The Sublime Macro Classifier introduces machine learning in MQL to detect malicious VBA macro attachments. Combining ML and MQL allows users to combine the model output with custom detection logic to surface what matters most while reducing the noise commonly associated with black-box ML approaches.

The classifier uses XGBoost to analyze VBA keywords, file metadata, and Oletools output to predict whether an attachment is likely to cause harm.

Use ml.macro_classifier to detect suspicious VBA macro attachments.

View rules that use this function

// detect malicious VBA macros in Office documents, high confidence
any(attachments, .file_extension in~ ("doc", "docm", "docx", "dot", "dotm", "pptm", "ppsm", "xlm", "xls", "xlsb", "xlsm", "xlt", "xltm", "zip")
    and ml.macro_classifier(.).malicious
    and ml.macro_classifier(.).confidence in ("high")
)

// detect malicious VBA macros in Office documents, low or medium confidence
any(attachments, .file_extension in~ ("doc", "docm", "docx", "dot", "dotm", "pptm", "ppsm", "xlm", "xls", "xlsb", "xlsm", "xlt", "xltm", "zip")
    and ml.macro_classifier(.).malicious
    and ml.macro_classifier(.).confidence in ("low", "medium")
)

ml.nlu_classifier

ml.nlu_classifier(input: str) -> NluResult

Natural Language Understanding, or NLU, provides users with a machine learning service to analyze text-based content. The service has two primary capabilities:

• Email Classification
• Named Entity Recognition

Email Classification

The Email Classification component takes a body of text as input and provides Intents and/or Tags.

Intent
Intents are top-level categories describing common language attackers use to carry out phishing attacks.

Tags
Tags are subcategories that provide additional context for financial-themed phishing attacks. The service returns the following values:

Example Usage

type.inbound
and any([body.plain.raw, body.html.inner_text], 
        any(ml.nlu_classifier(.).intents,
            .name == "bec" and .confidence == "high"
        )
)
// first-time sender
and (
  (
    sender.email.domain.root_domain in $free_email_providers
    and sender.email.email not in $sender_emails
  )
  or (
    sender.email.domain.root_domain not in $free_email_providers
    and sender.email.domain.domain not in $sender_domains
  )
)

Entity Recognition

Named Entity Recognition (NER) identifies, tags, and extracts important keywords within a body of text. Users can leverage this output to determine if an email contains language commonly associated with urgency, requests, or financial matters. The available entities are listed below:

Example Usage

type.inbound
and sender.display_name in~ $org_display_names
and any(ml.nlu_classifier(body.current_thread.text).entities, .name == "urgency")
and any(ml.nlu_classifier(body.current_thread.text).entities, .name == "request")

Considerations

It is important to remember that the NLU engine only looks at text. Because of this, it needs additional context to be an adequate detector. For example, attackers may craft an email that looks the same as a password reset for your favorite social network. The NLU engine would classify the text as cred_theft, but it would also do the same for a legitimate password reset email. But pairing it with a First-Time/Unsolicited Sender or LinkAnalysis provides the necessary context to make an effective detector.

Network Analysis

network.whois

network.whois(domain: Domain) -> WhoisOutput

network.whois performs a WHOIS lookup for domain registration on the .root_domain field of a Domain. It returns the domain age, registrar information, and timing information about the age of the registration record and when it was retrieved.

This function can be used to identify newly registered domains, by searching for domain age or if a domain is not found. Lookups are performed against Sublime's WHOIS service, which may be delayed by ~24 hours. Since new domains have a slight delay, searching for .found == false will identify both unregistered and newly registered domains. For some detections, the .found == false could be high enough signal.

View rules that use this function

network.whois(sender.email.domain).found == false or
network.whois(sender.email.domain).days_old <= 7

any(body.links, network.whois(.href_url.domain).days_old <= 14)

Profiling with historical context

📘

Behavior of historical functions

The result of historical functions is always relative to the time of the message that is being evaluated. During live processing, this means the latest possible information is available. However, during a backtest, these functions only take into account messages that are seen prior to that point in time. If there's not enough data, some fields like .prevalence may be "unknown". This behavior ensures that during a backtest there's never access to "future" data, which would lead to incorrect results and a false sense of confidence in the efficacy of a rule.

Results are typically and deliberately delayed by several hours, so that the prevalence of a sender can remain as"new"for approximately 8-12 hours.

profile.by_sender

profile.by_sender() -> SenderProfile

profile.by_sender uses previously ingested inbound messages to build a profile for messages received from a matching Sender. This profile captures information like the .prevalence of the sender domain within your environment to assess how common or uncommon it is across messages. It also captures information about flagged messages, such as false positives or true positives.

For the profile.by_sender function, the list $free_email_providers is used to determine whether a sender means a matching email or domain. If the value of sender.email.domain.domain is in $free_email_providers, then sender.email.email is used to determine a matching Sender. Otherwise, all messages with a matching sender.email.domain.domain are considered to be from the same Sender. This ensures that for profile.by_sender, a matching Sender covers messages from an organization, instead of an individual.

Using profile.by_sender() to find a first-time sender:

type.inbound
and profile.by_sender().prevalence == "new"

Using lists do find a first-time sender is the same but more verbose:

type.inbound
and (
  (
    sender.email.domain.root_domain in $free_email_providers
    and sender.email.email not in $sender_emails
  )
  or (
    sender.email.domain.root_domain not in $free_email_providers
    and sender.email.domain.domain not in $sender_domains
  )
)

To check against the historical reputation for a sender, check whether a sender has sent at least 1 message flagged as malicious or spam but no confirmed false positives.

type.inbound
and profile.by_sender().any_messages_malicious_or_spam
and not profile.by_sender().any_false_positives

// Additional logic on the suspicious sender.
and ...

Two more sender profile functions: profile.by_sender_domain and profile.by_sender_email exist if the automatic switching between email and domain is not preferred.

profile.by_sender_domain

profile.by_sender_domain() -> SenderProfile

profile.by_sender_domain uses previously ingested inbound messages to build a profile for messages received from a matching sender.email.domain.domain.

type.inbound

// filter by first-seen domains or anomalous domains in your environment
and profile.by_sender_domain().prevalence in ("outlier", "new")

// scrutinize PDF attachments, for example
and any(attachments, .file_extension == "pdf" and ...)

profile.by_sender_email

profile.by_sender_email() -> SenderProfile

profile.by_sender_domain uses previously ingested inbound messages to build a profile for messages received from a matching sender.email.email.

type.inbound

// filter by first-seen or anomalous email addresses in your environment
and profile.by_sender_email().prevalence in ("outlier", "new")

Together, profile.by_sender_domain and profile.by_sender_email can be used to tell when a domain is common but the sending email address is new:

type.inbound

// not a free email provider
and sender.email.domain.domain not in $free_email_providers

// domain is common in your environment
and profile.by_sender_domain().prevalence == "common"

// but this is the first time you've received messages from this sender
and profile.by_sender_email().prevalence == "new"