Scripting • Re: Address lists downloader (DShield, Spamhaus DROP/EDROP, etc)

I found this script on the forum. It works OK on my hEX S running 7.13.2.
The only change I've made was to concentrate all entries on a single "blacklist" and select the entries via the comment field.

MH

Code:

:global readfile do={    :local url        $1    :local thefile    ""    :local filesize   ([/tool fetch url=$url as-value output=none]->"downloaded")    :local maxsize    64512 ; # is the maximum supported readable size of a block from a file    :local start      0    :local end        ($maxsize - 1)    :local partnumber ($filesize / ($maxsize / 1024))    :local reminder   ($filesize % ($maxsize / 1024))    :if ($reminder > 0) do={ :set partnumber ($partnumber + 1) }    :for x from=1 to=$partnumber step=1 do={         :set thefile ($thefile . ([/tool fetch url=$url http-header-field="Range: bytes=$start-$end" as-value output=user]->"data"))         :set start   ($start + $maxsize)         :set end     ($end   + $maxsize)    }    :return $thefile}{/ip firewall address-list:local update do={ :global readfile :put "Starting import of address-list: $listname"  :put "Deleting all Dynamic enties in address-list: $listname"  :if (heirule != null) do={:put "Using as extra filtering: $heirule"}  :if ($heirule = null) do={:set $heirule "."}  :local n 0; # counter   # remove the current list completely :do { /ip firewall address-list remove [find where comment=$description dynamic]} on-error={};### line replaced ###  :local data ([:tool fetch url=$url output=user as-value]->"data")   :local data [$readfile $url]   :put "Imported file length $[:len $data] bytes"     :while ([:len $data]!=0) do={        :local line [:pick $data 0 [:find $data "\n"]]; # create only once and checked twice as local variable       :if ($line~"^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}" && $line~heirule) do={        :set $n ($n+1)         :do {add list=$listname address=[:pick $data 0 [:find $data $delimiter]] comment=$description timeout=$timeout} on-error={};       }; # if IP address && extra filter if present      :set data [:pick $data ([:find $data "\n"]+1) [:len $data]]; # removes the just added IP from the data array     }; # while :put "Completed importing $listname added/replacing $n lines."}; # do$update url=("https://" . "lists.blocklist.de/lists/all.txt") delimiter=("\n") listname=blacklist description=BlockDE timeout=1d$update url=("https://" . "www.dshield.org/block.txt") delimiter=("\t") listname=blacklist description=DShield timeout=1d$update url=("https://" . "iplists.firehol.org/files/firehol_level2.netset") delimiter=("\n") listname=blacklist description=FireHOLL2 timeout=1d$update url=("https://" . "view.sentinel.turris.cz/greylist-data/greylist-latest.csv") delimiter="," listname=blacklist description=GreyList timeout=1d heirule=http|smtp$update url=("https://" . "www.spamhaus.org/drop/drop.txt") delimiter=" ; " listname=blacklist description=SpamHaus timeout=1d$update url=("https://" . "sslbl.abuse.ch/blacklist/sslipblacklist.txt") delimiter=("\r") listname=blacklist description=SSLBL timeout=1d}

Statistics: Posted by RSE — Sat Mar 02, 2024 3:50 pm