HTTrack Website Scraper

Scrape using HTTrack (https://www.httrack.com/page/2/en/index.html )

 Every January, May and September the Registrar’s Office copies and archives certain RRU website pages; our policies, our academic regulations and all of the program information.

Due to upgraded Akamai bot policies and the age of HTTrack, specific configurations are required, particularly for the Programs. Presently this scan is locked to my IP address in Akamai (Tim Bugler)

Program Scan rules:

• -%u

• -*

• +*www.royalroads.ca/explore-programs-credential*

• +*www.royalroads.ca/programs/*

• +*www.royalroads.ca/course/*

• -*www.royalroads.ca/programs/*/*/*

• -*?tab=overview*

• -*?tab=meet*

• -*?tab=faculty*

• -*?tab=financial*

• -*bm-verify*

• -*?utm_*

• -*?session*

Academic Scan Rule:

• +.pdf

• +*www.royalroads.ca/sites/default/files*

• --disable-security-limits

Policies Scan Rules:

• -*

• +*www.royalroads.ca/policies/*

• +*http://www.royalroads.ca/policies?*

• +*www.royalroads.ca/sites/default/files/*

• --disable-security 

Common settings regardless of scan rules:

Browser ID ():

• WinHTTrack Website Copier 3.49-2/RRU Scrape Internal

Limits:

Open

Spider:

Open

Expert Only:

Open