We have put considerable effort into making our crawler as versatile as possible, to minimize the need for optimizations on your end. However, there are some guidelines that you can follow that are generic to web crawling:
- Use a browser such as Lynx to see how your site behaves through the eyes of a web crawler. Features such as cookies, sessions, DHTML, jscript can make the user experience very rich however can cause issues for web crawlers.
- Publish an RSS feed for all your properties.
- Publish a single GET or POST request link somewhere on your site that displays all your properties.
- If your website has a results list that is paginated (results 1-10.11-20 and so forth) do not obfuscate them in javascript form links, rather publish them as simple HTML GET requests.
- Allow Globrix to crawl your website without the necessity of session IDs or arguments.
- Allow people to link back to your website without the necessity of a session.
- Publish a site map HTML page that links to lists of properties for all the areas you cover.
Technical guide 
- 
