Search is part of the fundamental infrastructure of the Internet. And, it is currently broken.

Why is it broken? It is broken for the same reason that proprietary software is always broken: lack of freedom, lack of community, lack of accountability, lack of transparency. Here, we will start to change all that.

Grub started back in 2000 with a simple concept of distributing part of the search process pipeline: crawling. In a way, we were a bit ahead of our time, but our intention then was what it is now. We want to help fix search.

Now, with the help of community members and Open Source developers our time has come again. Come be part of something greater. Come help us change the World.

Grub NG Python client 0.4.3 released

Submitted by Sealabs on Mon, 06/29/2009 - 19:52.

A new version of the GrubNG python client is available. This version supports the new dispatcher.

Changelog since version 0.4:

  • Added variable to control the number of uploader threads
  • Changed verbose mode to default. Use -q flag to disable it
  • Added debug flag to save workunits and arc files on errors
  • Reupload the arc file in case of a network error (10 times)
  • File close() on everything to prevent 'Too many open files' errors
  • Changed to new grub.org dispatcher address
  • Fixed compressed responce data bug
  • Fixed bug about custom headers
  • Fixed bug about incomplete arc files treated as complete

Direct Download
Happy crawling ;)

Grub C# client 0.8.8 released

Submitted by thindil on Mon, 06/29/2009 - 09:06.

New version of Grub Next Generation C# client is available. Changes since last release (0.8.7):

- update Spanish and Galician translations (author: Víctor Hermida Prada)
- added French translation (author François Blondel)
- fixed option to run/pause selected crawler
- fixed bug with crash client during start
- fixed bug on crash client when sitemap file not exist #401: Grub bug in Windows Xp
- fixed bug with not re-send .arc file on upload failure
- added support for META ROBOTS tag
- small code optimization

Grub Next Generation upload server 0.6 released

Submitted by thindil on Fri, 06/19/2009 - 15:51.

Second beta version of Grub Next Generation C# upload server is available. Changes since last release (0.5):

- code cleanup and refactoring
- move uploading and checking .arc files to separate program
- fixed bug with count connected clients
- fixed server answer on invalid POST request
- fixed bug with crash server on copy not existing file
- fixed bug with crash server on unfinished files download
- fixed bug #377: Connection gets reset instead of closed after disk full error
- added simple connections log
- added option to move .arc files on disk (not only to Hadoop)

Start work on index

Submitted by thindil on Thu, 06/18/2009 - 20:02.

Today we start works on indexing crawled by Grub volunteers data. First effects of our work you can find at: http://soap.grub.org

This is very basic web search interface with small index (at this moment around 166 000 of URL's). For indexing we use Solr. All other code (current web interface, converter from .arc files to solr xml upload) you can find in our repository in perl directory.

Grub C# client 0.8.7 released

Submitted by thindil on Mon, 06/15/2009 - 08:34.

New version of Grub Next Generation C# client is available. Changes since last release (0.8.6):

- update Spanish translation (authors: DiegoJ, Dennis Tobar)
- update German translation (authors: HennR, noxin)
- update Galician translation (author: Victor Hermida Prada)
- added option to enable/disable creating sitemaps during crawling
- fixed bug with not quit from program when paused
- fixed bug with crash client when antivirus move or delete .arc file #376: Grub crashed after Avast moved a Trojan to quarantine folder

1234567next ›last »
Syndicate content