Crawler information

Panhispanic Media Research Bot

Name

Crawler: Panhispanic Media Research Bot
User-Agent: PanhispanicMediaResearchBot/<version> (+https://coprepan.hispanistica.com/crawler/; felix.tacke@uni-marburg.de)
Token for robots.txt: PanhispanicMediaResearchBot

Operator

Marburg University
CO.PRE.PAN research project

Purpose

Scholarly collection of publicly accessible Spanish-language press material for corpus-linguistic research: non-commercial scientific text and data mining by a university.

The crawler is under development. In an initial test phase it visits only a small number of selected websites.

Behaviour

It requests public, unauthenticated HTTP(S) resources only.
It reads robots.txt before anything else on a site, and what the file says for this crawler is recorded with every request.
For non-commercial scientific text and data mining of publicly accessible pages, the project does not treat every Disallow line as an absolute exclusion. Where a page that is openly reachable without any login is requested although a Disallow rule applies to it, that fact, the rule and the reason are recorded with the request. This is the policy of a research project, stated here so that you know it; we do not claim that it follows the Robots Exclusion Protocol in this point.
It does not bypass authentication, paywalls, CAPTCHAs, bot challenges, IP or rate-limit blocks or any other technical access control, and it never disguises itself. If a server refuses access (for example with status 401, 403 or 429), the refusal is recorded and the crawler stops requesting that site.
A direct opt-out is honoured: if you write to us, requests to your site stop and a person answers you.
It makes one request at a time to a website and keeps a conservative pace, at present at least ten seconds between two requests to the same site. A Crawl-delay in your robots.txt is honoured as a minimum pause where it is given as a plain number. A site that asks for a very long pause is not crawled at all.
It respects Retry-After.
It uses conditional requests (If-None-Match, If-Modified-Since) where appropriate, to avoid transferring unchanged pages again.
Preserved source material is kept as research evidence. It is not automatically redistributed publicly.

Contact and opt-out

Felix Tacke
Marburg University
felix.tacke@uni-marburg.de

Website operators may contact us at this address regarding crawler behaviour, exclusion or opt-out requests. The most reliable ways to keep this crawler away are a message to this address or an ordinary access restriction on your server; both are always respected.

Información en español

Panhispanic Media Research Bot es el robot de rastreo del proyecto de investigación CO.PRE.PAN (Corpus de Prensa Panhispánica) de la Universidad de Marburgo (Marburg University). Su finalidad es la recopilación científica de prensa en español accesible públicamente para la investigación lingüística de corpus.

Solo solicita recursos HTTP(S) públicos y sin autenticación.
Lee robots.txt antes que nada y registra lo que el archivo indica para este robot. Para la minería de textos y datos con fines de investigación científica no comercial sobre páginas de acceso público, el proyecto no considera cada línea Disallow como una exclusión absoluta; cuando se solicita una página a la que se aplica una regla Disallow, ese hecho y su motivo quedan registrados.
No elude la autenticación, los muros de pago, los CAPTCHA, los desafíos anti-bot, los bloqueos ni otros controles técnicos de acceso, y nunca oculta su identidad. Si un servidor deniega el acceso, el robot deja de solicitar ese sitio.
Una solicitud directa de exclusión se respeta: las solicitudes a su sitio se detienen y una persona le responde.
Realiza una sola solicitud a la vez por sitio web y mantiene un ritmo prudente; respeta Retry-After y, cuando es un número simple, Crawl-delay.
El material conservado es evidencia de investigación y no se redistribuye públicamente de forma automática.

Los administradores de sitios web pueden escribirnos a felix.tacke@uni-marburg.de para consultas sobre el comportamiento del robot, la exclusión de su sitio o solicitudes de baja.