Dan Clowry's Blog

Using Checkpoint Avanan and Microsoft 365 Quarantine

Daniel Clowry — Tue, 12 Dec 2023 10:20:25 GMT

Throughout this article I'll be refering to Avanan as "Checkpoint" now that it has been acquired by Checkpoint and merged into their product line.

Last updated 12 December 2023
Checkpoint system version PROD_2023_w49_6

Intro

I work for an Australian MSP servicing SMB clients. Earlier this year we began replacing our previous security email gateway service with Checkpoint Protect/Avanan throughout our client setups.

One issue we struggled with during our initial deployment was handling quarantined messages. Most SEG solutions require you to replace your domain's MX record with the SEG's mailserver and create transport rules in Exchange to completely bypass 365's spam filtering. Checkpoint on the other hand works in addition to Exchange's built in protection by scanning emails using APIs. This means your emails are protected with an additional layer of security through Exchange Online Protection/Defender for Office 365 but also means you're now managing two sets of filtering policies and importantly, two sets of quarantine systems.

Having to deal with two sets of quarantine systems adds unnecessary complexity for end users as well as techs. Users would receive digests from both systems, and would need to deal with two different UIs for viewing & releasing email. Similarly, techs would need to check two systems when investigating quarantined emails and we needed to separately integrate both Checkpoint and 365 quarantine into our ticketing system. Luckily since June 2023 Checkpoint has a new "Unified Quarantine" feature that combines emails quarantined by Microsoft and Checkpoint into a single end-user digest as well as a single portal for techs to action quarantined items.

A Checkpoint quarantine digest. The report contains emails that have been quarantined by both Microsoft and Checkpoint.

Enabling Unified Quarantine notifications

Unified Quarantine digests will contain a list of emails that have been quarantined by both Checkpoint and Microsoft since the last digest. Users can then release, or request release quarantined emails per your Checkpoint Office 365 Mail policy (for emails quarantined by Checkpoint) and the Microsoft release options policy configured in the steps below (for emails quarantined by Microsoft.)

From your Checkpoint tenant, expand Security Settings > User Interaction > Restore Requests.
Enable End User Quarantine Report.
If desired, tick "Include spam emails that are sent to the Junk folder", "Allow end users to generate a quarantine report on demand", and "Stop sending immediate quarantine notification emails". I normally enable "Allow end users to generate a quarantine report on demand" while leaving the other two disabled.
Expand the Scheduling section and create your desired schedule. This could be once a day, once every hour, 10am & 4pm, etc.
Expand Recipients and select All Users.
Expand Sender and set the following options:
- Friendly-From name: This sets the From header display name. You can set whatever name you want or leave it blank. If left blank the full sender address will be displayed in the user's email client.
- From address: Change to Custom and set the address as no-reply@quarantine.[yourdomainhere] e.g. no-reply@quarantine.danclowry.com. The section "High-Confidence Phish & Secure By Default" further down this article explains why the "quarantine" subdomain is used rather than leaving the address as the default no-reply@[recipient domain].
- Reply-to address: Leave as Same as From address
Expand Emails quarantined by Microsoft and tick "Daily quarantine report should include emails quarantined by Microsoft"
Set the release options for Microsoft quarantined emails. Below are the default actions:
- Malware: Can request a restore (admin needs to approve)
- High Confidence Phishing: Can request a restore (admin needs to approve)
- Phishing: Can request a restore (admin needs to approve)
- High Confidence Spam: Can restore on their own
- Spam: Can restore on their own
- Bulk: Cannot restore
- Transport Rule: Cannot restore
Click Save and Apply to save the notification settings

Checkpoint will now begin sending quarantine digests to end users per your configured schedule. Job done right? Unfortunately not...

High-Confidence Phish & Microsoft 365 Secure by Default

After configuring unified quarantine notifications for our clients, I quickly noticed that Checkpoint's digest emails were being quarantined by 365 as "high confidence phish". I initially tried creating an Exchange transport rule to set the SCL to -1 for any emails coming from a Checkpoint IP with sender no-reply@[customerdomain] to bypass EOP spam filtering. However even after creating the transport rule, digests were still getting quarantined by 365 as high confidence phish.

After further digging, I found that this was because of 365's "secure by default" feature where Microsoft will ignore common EOP overrides for emails detected as malware or high confidence phishing. In Microsoft's own words:

Because Microsoft wants to keep our customers secure by default, some tenants overrides aren't applied for malware or high confidence phishing. These overrides include:

Allowed sender lists or allowed domain lists (anti-spam policies)

Outlook Safe Senders

IP Allow List (connection filtering)

Exchange mail flow rules (also known as transport rules)

Microsoft will only enable secure by default when the MX record for your domain is the default protection.outlook.com record. Because Checkpoint uses APIs rather than a traditional MX record replacement, secure by default is switched on and cannot be disabled.

So how can you stop 365 from quarantining Checkpoint's digests? Microsoft lists four ways you can override secure by default:

Phishing simulations: Simulated attacks can help you identify vulnerable users before a real attack impacts your organization. To prevent phishing simulation messages from being filtered, see Configure third-party phishing simulations in the advanced delivery policy.

Security/SecOps mailboxes: Dedicated mailboxes used by security teams to get unfiltered messages (both good and bad). Teams can then review to see if they contain malicious content.

Third-party filters: Secure by default only applies when the MX record for your domain is set to Exchange Online Protection (contoso.mail.protection.outlook.com). If it's set to another service or device, it's possible to override Secure by default with a Transport Rule to bypass all spam filtering. When Microsoft detects messages as High Confidence Phish with this rule in place, they still deliver to the Inbox.

False positives: To temporarily allow certain messages that are still being blocked by Microsoft, use admin submissions. By default, allow entries for domains and email addresses, files, and URLs exist for 30 days. During those 30 days, Microsoft learns from the allow entries and removes them or automatically extends them. By default, allow entries for spoofed senders never expire.

Security/SecOps mailboxes won't work as that'll just redirect everyone's quarantine digests to a single mailbox (as well as any other actual spam filtered by EOP), changing the MX record isn't possible due to how Checkpoint works, and submitting the quarantine digests as false positive detections in Defender would work OK for a single deployment, but doesn't scale when you're managing dozens or hundreds of tenants.

Configuring a phishing simulation is what we'll use. This is basically the same as a spam bypass transport rule and will set Exchange's SCL to -1 for Checkpoint's digests.

Configuring a Phishing Simulation

Before you can create the phishing simulation you'll need to figure out what IP address your Checkpoint digest emails are sent from so you can add it to the phishing simulation policy. Checkpoint specifies its mailserver IP addresses in its spfa.cpmails.com SPF record. At the time of writing, the SPF record contains 12 IPv4 addresses. Microsoft only allow up to 10 addresses to be added to a phishing simulation policy so you can't just add all 12 addresses to your policy. Luckily, Checkpoint seems to send all its notification emails from a single IP (likely based on your tenant's configured location). For all of our tenants in Australia, Checkpoint sends its digests from the 13.211.69.231 IP address (which has a PTR record of au.cloud-sec-av.com). Check your previous Checkpoint digests to figure out what IP address Checkpoint uses on your tenant.

A quarantine report with its headers shown on the right. Note the sender IP 13.211.69.231. All of our quarantine digests are sent from this IP. The IP is also present in the spfa.cpmails.com SPF record.

Once you've found what IP Checkpoint uses, you're ready to create a phishing simulation policy. To create the policy:

Login to your 365 tenant and go to the Microsoft 365 Defender portal.
Go to Policies & rules > Threat policies > Advanced delivery > Phishing simulation.
Click Add to open the Add Third Party Phishing Simulations flyout.
Configure the phishing simulation options:
- Expand the Domain option and enter the quarantine.[yourdomainhere] domain you set in Checkpoint.
- Expand the Sending IP option and enter the IP Checkpoint sends its emails from.
- Leave Simulation URLs to allow blank as this isn't required for emails.
Click Add to save the policy.

Exchange will now bypass its spam filtering for all emails sent from the quarantine.[yourdomainhere] address if they were sent through your specified Checkpoint IP. If an email does not match both the domain AND IP conditions, Exchange will treat it as a regular email and it will be subject to EOP filtering.

Protecting the quarantine subdomain from spoofing

It is important to configure an SPF and DMARC record for the quarantine subdomain to prevent an attacker from spoofing it either internally or externally.

Since July this year Exchange Online has begun honoring domain DMARC policies by default. I.e. if a domain's DMARC policy has a value of p=reject, Exchange will return an NDR to the sender and refuse to deliver the message. If a domain has a DMARC policy of p=quarantine, EXO will move the message into 365 quarantine.

We can create an SPF record alongside a DMARC record for the quarantine subdomain to ensure that Exchange will always reject emails unless they were sent from one of the Checkpoint servers.

Add the following two TXT records to your domain's zone:

Type	Name	Value	TTL
TXT	quarantine	v=spf1 include:spfa.cpmails.com -all	Default
TXT	_dmarc.quarantine	v=DMARC1; p=reject; aspf=s; adkim=s;	Default

The SPF record informs recipient servers that the Checkpoint mail servers are the only permitted senders for the domain. The DMARC record then informs recipient servers that they should reject all messages that fail SPF validation.

Now if a third-party tries to spoof the quarantine subdomain, the message will be immediately rejected by Exchange with the response:

550 5.7.509 Access denied, sending domain quarantine.danclowry.com does not pass DMARC verification and has a DMARC policy of reject.

Conclusion

Now that you've configured Checkpoint Unified Quarantine your users will only have to worry about checking a single report to view their quarantined messages.

How to Install Youtube-DL on Windows

Daniel Clowry — Wed, 17 Jul 2019 02:04:00 GMT

Youtube-DL is a great little program that allows you to download videos from almost every major video website on the internet. This tutorial will show you how to download, install, configure and use Youtube-DL for Windows.

Prerequisites:

Basic knowledge on how to open and use Command Prompt
Microsoft Visual C++ 2010 Redistributable Package (x86)

Download and Installation

Installing Youtube-DL

We'll start off with download and installing Youtube-DL. Go to the releases page of the Youtube-DL Github repository. Scroll down until you find the list of assets, click on the file called "youtube-dl.exe" to begin downloading Youtube-DL. You can download the file to any location except for System32. I recommend creating a separate folder solely for Youtube-DL as it makes it easier to configure Youtube-DL later on. For this tutorial, I have downloaded the file to D:\Programs\Other Programs\Youtube-DL.

Now that you've downloaded Youtube-DL, you've probably noticed that nothing happens when you try to double-click to run the program. This is because Youtube-DL is a command-line program, it can only be used from a command line interface such as Command Prompt.

However, if you open Command Prompt and try to use the command youtube-dl you'll still get an error saying that the command is not recognised. To run Youtube-DL you'll need to either add it to your computer's PATH or cd into the directory where you downloaded Youtube-DL.

Add Youtube-DL to your PATH (Recommended)

Adding Youtube-DL to you PATH allows you to run the youtube-dl command from any command-line interface such as Command Prompt or PowerShell without needing to first cd into the directory where you installed Youtube-DL.

To add Youtube-DL to your PATH:

1. Copy the full path of your Youtube-DL install location - You can reveal the full path by navigating to wherever you downloaded Youtube-DL in Windows Explorer, clicking on the address bar and copying the text. Do not include the .exe in the path, it should only include directories (e.g. D:\Programs\Other Programs\Youtube-DL instead of D:\Programs\Other Programs\Youtube-DL\youtube-dl.exe)

2. Open Environment Variables - Open the System Properties menu by pressing Win+R and typing systempropertiesadvanced. From this menu, press the button labelled "Environment Variables...".

3. Select the Path variable - You can add Youtube-DL to either the user or system PATH. Adding it to the system PATH will allow any user to run the youtube-dl command without first cding into the install directory, whereas the user PATH will only apply to the currently logged in user. Note that setting it in the system PATH requires administrator permissions and youtube-dl.exe cannot be located in a user specific location (such as the Desktop or "My Documents"). For this tutorial, I will be using the system PATH.

If you want to add it to the user PATH, scroll through the first list of variables until you find one named Path. Double-click on it to open the editor.
If you want to add it to the system PATH, scroll through the second list of variables until you find one named Path. Double-click on it to open the editor.

There will most likely already be some stuff already in the Path. Do not delete any existing entries. They were most likely set by another program or by Windows and deleting them can cause programs not to work and can even cause Windows to become unbootable.

List of user and system variables on the left and the editor for the system PATH on the right

4. Add the Youtube-DL install path to Path - With the variable editor open and your Youtube-DL install path in the clipboard, press the "New" button from the right side of the editor. This will create a new entry to the bottom of the variable. Paste in your Youtube-DL path and press enter then press "OK" to save the changes.

5. Test it! - Now that you've added Youtube-DL to the PATH you should be able to open a new Command Prompt window and run youtube-dl. Now you can go to the section "Installing FFmpeg" to continue the installation!

If you're still getting an error saying that youtube-dl is not recognised, try the following:

Restart your computer to force a refresh of the PATH.
Double check that the path to Youtube-DL is correct. Make sure that youtube-dl.exe is not included in the path.
If you added Youtube-DL to the system PATH, try adding it to the user PATH (and vice versa).
From the PATH editor, select the Youtube-DL entry and try moving it up or down the list of PATH entries using the "Move Up" and "Move Down" buttons.

And if you still can't get it it work, leave a comment at the bottom of this article to see if someone can help you.

Run Youtube-DL from the Install Directory

If you only want to run Youtube-DL occasionally and don't want it cluttering your PATH, you can instead run it directly from the install folder. Open Windows Explorer and navigate to the folder you downloaded Youtube-DL to. Click on the address bar in Windows Explorer and type cmd to open Command Prompt in the current folder. Now you can simply use youtube-dl to run the program.

Installing FFmpeg

While FFmpeg isn't required, it gives you access to more features of Youtube-DL. Most notably, it allows you to download Youtube videos higher than 720p and allows you to combine a specific audio and video track together as one file.

Installing FFmpeg is pretty much the same as Youtube-DL. You can download Windows binaries for FFmpeg from FFmpeg maintainer Gyan Doshi's website — download the "git-full" binary if you're not sure what build to pick. This will begin downloading a 7z file containing the latest FFmpeg build (you will need 7-Zip to extract the file). Open the archive and extract the contents of the ffmpeg-20xx-xx-xx folder to a location of your choosing. For this tutorial, I have extracted it to D:\Programs\Other Programs\ffmpeg.

With FFmpeg extracted, you'll now need to add it to your PATH. The steps are the same as with Youtube-DL. However, instead of copying the base FFmpeg install path, copy the path from the bin folder (e.g. D:\Programs\Other Programs\ffmpeg\bin instead of D:\Programs\Other Programs\ffmpeg\).

Note that the path for FFmpeg includes the "bin" folder

You can verify that FFmpeg is set in the path by opening Command Prompt and run ffmpeg.

Configuring Youtube-DL

Now that you've successfully installed Youtube-DL, you'll probably want to customise it to your liking using some of the hundreds of options available. But typing out the same options every time you run Youtube-DL is tedious. This is where the configuration file comes in. The options set in the configuration file are run every time Youtube-DL is run. However, the config can be disabled at runtime using the --ignore-config flag. Options set at runtime will also have precedence over those set in the config.

Creating a config file is very straightforward. For Windows, simply create a file called config.txt at %APPDATA%\youtube-dl\. You set options in the config much like how you set them at runtime. You can comment out lines using the hash symbol (#).

I recommend using the following config as a starting point:

# Set date at current date instead of video's original date
--no-mtime

# Save to current user's videos folder using video title
# See https://github.com/ytdl-org/youtube-dl#output-template for all naming options
-o "%USERPROFILE%\Videos\Youtube\%(title)s.%(ext)s"

See the Youtube-DL list of options and list of output options for configuring Youtube-DL.

Using Youtube-DL

So you've finally installed and configured Youtube-DL to your liking, you're probably wondering how you use it. This section gives some examples and tips on how to download videos using Youtube-DL.

Basic Usage

The easiest way to download a video is by calling youtube-dl followed by the video URL.

youtube-dl https://www.youtube.com/watch?v=tO01J-M3g0U

This will download the requested video using the highest available quality. If FFmpeg is installed, it will download the highest quality video and highest quality audio separately and combine them into a single file. Otherwise, it will instead try to download the highest quality file that contains both video and audio.

Format Selector

But what if you want to download the video at a specific quality? Or what if you want to only download audio? This is where the format selector comes in.

You can list all the available formats for a video by using the -F flag (note the uppercase F) or --list-formats.

youtube-dl https://www.youtube.com/watch?v=tO01J-M3g0U -F

[youtube] tO01J-M3g0U: Downloading webpage
[youtube] tO01J-M3g0U: Downloading video info webpage
[info] Available formats for tO01J-M3g0U:
format code  extension  resolution note
249          webm       audio only tiny   56k , opus @ 50k (48000Hz), 918.04KiB
250          webm       audio only tiny   71k , opus @ 70k (48000Hz), 1.19MiB
140          m4a        audio only tiny  130k , m4a_dash container, mp4a.40.2@128k (44100Hz), 2.39MiB
251          webm       audio only tiny  140k , opus @160k (48000Hz), 2.37MiB
394          mp4        256x144    144p   79k , av01.0.00M.10.0.110.09.16.09.0, 30fps, video only, 1.24MiB
278          webm       256x144    144p   97k , webm container, vp9, 30fps, video only, 1.59MiB
160          mp4        256x144    144p  111k , avc1.4d400c, 30fps, video only, 1.20MiB
395          mp4        426x240    240p  182k , av01.0.00M.10.0.110.09.16.09.0, 30fps, video only, 2.27MiB
242          webm       426x240    240p  227k , vp9, 30fps, video only, 3.20MiB
133          mp4        426x240    240p  245k , avc1.4d4015, 30fps, video only, 2.45MiB
330          webm       256x144    144p60 HDR  250k , vp9.2, 60fps, video only, 3.69MiB
396          mp4        640x360    360p  336k , av01.0.01M.10.0.110.09.16.09.0, 30fps, video only, 4.26MiB
243          webm       640x360    360p  420k , vp9, 30fps, video only, 5.97MiB
331          webm       426x240    240p60 HDR  517k , vp9.2, 60fps, video only, 7.98MiB
397          mp4        854x480    480p  623k , av01.0.04M.10.0.110.09.16.09.0, 30fps, video only, 7.92MiB
134          mp4        640x360    360p  634k , avc1.4d401e, 30fps, video only, 6.97MiB
244          webm       854x480    480p  773k , vp9, 30fps, video only, 10.86MiB
332          webm       640x360    360p60 HDR 1094k , vp9.2, 60fps, video only, 17.37MiB
398          mp4        1280x720   720p60 1249k , av01.0.08M.10.0.110.09.16.09.0, 60fps, video only, 17.83MiB
135          mp4        854x480    480p 1297k , avc1.4d401f, 30fps, video only, 14.05MiB
247          webm       1280x720   720p 1545k , vp9, 30fps, video only, 22.19MiB
333          webm       854x480    480p60 HDR 2034k , vp9.2, 60fps, video only, 33.47MiB
136          mp4        1280x720   720p 2657k , avc1.4d401f, 30fps, video only, 27.20MiB
248          webm       1920x1080  1080p 2702k , vp9, 30fps, video only, 39.80MiB
302          webm       1280x720   720p60 2710k , vp9, 60fps, video only, 36.54MiB
298          mp4        1280x720   720p60 4071k , avc1.4d4020, 60fps, video only, 45.17MiB
303          webm       1920x1080  1080p60 4622k , vp9, 60fps, video only, 63.87MiB
334          webm       1280x720   720p60 HDR 4790k , vp9.2, 60fps, video only, 80.74MiB
137          mp4        1920x1080  1080p 4985k , avc1.640028, 30fps, video only, 49.39MiB
299          mp4        1920x1080  1080p60 6761k , avc1.64002a, 60fps, video only, 80.25MiB
335          webm       1920x1080  1080p60 HDR 7277k , vp9.2, 60fps, video only, 123.60MiB
271          webm       2560x1440  1440p 8874k , vp9, 30fps, video only, 119.07MiB
308          webm       2560x1440  1440p60 13422k , vp9, 60fps, video only, 181.85MiB
336          webm       2560x1440  1440p60 HDR 16900k , vp9.2, 60fps, video only, 297.56MiB
313          webm       3840x2160  2160p 18006k , vp9, 30fps, video only, 287.61MiB
315          webm       3840x2160  2160p60 26847k , vp9, 60fps, video only, 436.24MiB
337          webm       3840x2160  2160p60 HDR 29609k , vp9.2, 60fps, video only, 528.73MiB
43           webm       640x360    360p , vp8.0, vorbis@128k, 15.53MiB
18           mp4        640x360    360p  609k , avc1.42001E, mp4a.40.2@ 96k (44100Hz), 11.23MiB
22           mp4        1280x720   720p 1605k , avc1.64001F, mp4a.40.2@192k (44100Hz) (best)

Youtube-DL returns a lot of information but once you break it down it becomes a lot more manageable. The information is returned in a table format with seven columns and one format per row. Audio-only formats comes first, followed by video-only formats with the audio+video formats at the bottom. Formats are listed in ascending order by bitrate. The columns of the table are broken down as follows:

Format code - The code used when downloading a format
Extension - The extension of the downloaded format
Resolution - The resolution, framerate and additional information such as HDR
Bitrate - The bitrate of the format expressed in kilobits/second
Codec - Codec information, format framerate and whether the format is video/audio only
Size - The size of the format

Format Code	Extension	Resolution	Bitrate	Codec	Size
`337`	`webm`	`3840x2160 2160p60 HDR`	`29609k`	`vp9.2, 60fps, video only`	`528.73MiB`

To download specific specific formats of a video, use the -f (note the lowercase f) or --format flag. You can specify video only, audio only, or video + audio formats. Formats are selected using the format codes returned using the --list-formats flag.

youtube-dl https://www.youtube.com/watch?v=tO01J-M3g0U -f 299+251

[youtube] tO01J-M3g0U: Downloading webpage
[youtube] tO01J-M3g0U: Downloading video info webpage
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[download] Destination: C:\Users\Dan\Videos\Youtube\The World in HDR in 4K (ULTRA HD).f299.mp4
[download] 100% of 80.89MiB in 00:28
[download] Destination: C:\Users\Dan\Videos\The World in HDR in 4K (ULTRA HD).f251.webm
[download] 100% of 2.37MiB in 00:01
[ffmpeg] Merging formats into "C:\Users\Dan\Videos\Youtube\The World in HDR in 4K (ULTRA HD).mkv"
Deleting original file C:\Users\Dan\Videos\Youtube\The World in HDR in 4K (ULTRA HD).f299.mp4 (pass -k to keep)
Deleting original file C:\Users\Dan\Videos\Youtube\The World in HDR in 4K (ULTRA HD).f251.webm (pass -k to keep)

The above example will download the formats that match the codes 299 and 251. This corresponds to the 1080p 60FPS video and 140kb/s Opus audio stream. Those files are then combined into a single MKV using FFmpeg.

Downloading audio or video only follows the same process. Just specify the audio or video format.

youtube-dl https://www.youtube.com/watch?v=tO01J-M3g0U -f 299

Only download video (format code 299)

youtube-dl https://www.youtube.com/watch?v=tO01J-M3g0U -f 251

Only download audio (format code 251)

Youtube-DL also has some preset formats to download the best or worst audio/video available. These can be used in place of an exact format code.

best - Downloads the highest quality file containing both audio and video
worst - Downloads the lowest quality file containing both audio and video
bestvideo - Downloads the highest quality video-only file
worstvideo - Downloads the lowest quality video-only file
bestaudio - Downloads the highest quality audio-only file
worstaudio - Downloads the lowest quality audio-only file

Conclusion

Youtube-DL is an incredible program that allows you to download media from hundreds of websites. While I hope this tutorial has got you up and running with using Youtube-DL, I highly recommend checking out the README for Youtube-DL that contains much more detailed information on the available options for using and configuring Youtube-DL.

Changelog

2021-01-23: updated FFmpeg download link (thanks ElectroGaming)