Tiko-Crawler

The crawler for the Tiko search engine.

Build instructions

To build tiko-crawler, run make:

make

This will produce a binary file, ./tikoc.

How to use

Run ./tikoc.
You will be asked to enter a starting point. This is where tiko-crawler will begin to look for domains to add.
For now, tiko-crawler only fetches up to 500 domains. This is defined in include/crawl.hpp at line 11. It is planned to have this be configurable at some point.

Once the crawling process is done, tiko-crawler should generate ./sites.tdb. This file is the database file that Tiko will be able to use. To do so, rename sites.tdb to db.txt and move it in the root of a Tiko installation.

Differences from other crawlers

Unlike most crawlers, tiko-crawler only fetches a domain once.
For example, if you have two links to the same website:

https://example.com/somefile
https://example.com/

tiko-crawler will only register and fetch example.com once. Some people might find this limiting since you cannot look for specific pages in a website, however I find this beneficial as it aligns more with the goals of Tiko.