Standards |
 |
JSpider isn't just a piece of software that is created the way we think it should be created,
or that lives by it's own rules.
we adhere to some well-established standards to ensure the usability and interoperability
of our software.
RFC 2616: HTTP/1.1 protocol standard
Because JSpider is a web spider, it will generate HTTP requests. We strictly adhere to the HTTP
standard as defined by the RFC 2616.
This includes the support for cookies, user agent headers, redirects, ...
Robots.txt: the robots exclusing standard
By default, JSpider is configured to be a well-behaving web robot, that obeys the instructions
of the webmaster in the robots.txt file, according to the robots.txt
robot exclusion standard.
|
