Uploading failed after crawling

Project:Grub Next Generation Python Client
Έκδοση:0.2-testers-wanted
Component:Code
Κατηγορία:bug report
Priority:normal
Assigned:Unassigned
Κατάσταση:closed
Περιγραφή

This is my second attempt at posting this, so please forgive me if I miss out anything.
I was running grub 0.2
on windows XP SP3
using python 2.5.2
I ran grub with the following flags:
-v -t 1 -u suicideducky -p

The uploading of the arcfile failed, and the message it gives me indicates that the reponse.status is neither 200 nor 500, infact it appears to be 401. Which as far as I know (in http) means an authentication error, I am not sure why this would happen as the connection used to upload the arcfile passed no authentication data (USERNAME or PASSWORD).

I have a screenshot of the command session, but it does not give much more information.
It behaved this way constantly over many runs over half and hour or so.
I will try again tomorrow incase it is a server or internet problem.

Chris.

#1

I downloaded and tested, with the same flags on my Archlinux system and things worked fine.
I am, for now, assuming this was a issue with either windows xp or the internet connection at the time. I will post more details as I find them.

#2

I'm having the same problem with uploading (NG C# client 0.8.3; Windows XP SP3).

#3

A few hours after this incident, I tested grub again from a different computer, this time running Linux, and it had multiple successes with no sign of any problems. Same flags, same internet connection, same username and password.

#4

Still getting the same problem.

#5

@Suicideducky, Dr. Breznjev
You get only 401 errors, or some 401 errors?

#6

To clarify, it says uploading failed and prints the following
code: 401 response:
followed by some whitespace, I will attach the picture I have of it later on tonight.

[edit]Origionally, on my windows PC I was gettin ONLY 401 errors, now on my Linux machine I am getting SOME 401 errors, will test furhur on my home PC to see if I get some non errors.

Best of luck,
Chris

[EDIT]
here is what it says exactly
"""[Mon Jan 12 11:45:49 2009] Thread 0 is fetching Work Unit
[Mon Jan 12 11:45:52 2009] Thread 0 is checking Work Unit
[Mon Jan 12 11:45:52 2009] Thread 0 is crawling
[Mon Jan 12 12:01:02 2009] Thread 0 is packing
[Mon Jan 12 12:01:02 2009] Thread 0 is uploading
[Mon Jan 12 12:01:08 2009] Thread 0 upload FAILED
Code: 401 Response:

"""
Minus the quotation marks of course, this time it appeared on my Linux machine (python 2.6.2) but after an already successfull upload, so I had one success and one failure.

[EDIT]
been running grub all day, 4 threads this time,
over the course of the day have had 5 failed uploads, all same error code. And about 40+ successes.

#7

New server version released (still in testing) *should* fix this problem.
I am also adding a feature to grub to mitigate this error.

Thanks again,
Chris.
[EDIT] The next server rollout should mitigate this

#8

When I try to upload, I get a message of an internal server error after which the program crashes. After restarting, the uploading fails and the crawl has to be redone. Besides that, in debug mode there are two warnings of a symbol file (Grubng.exe.mdb and grubng-sharp.dll.mdb resp.) with an incorrect version (39 instead of 50.0)

I added the error log.

#9

This is about C# version, so i try answer ;)
I have question: did you try change server with workunits address (how this is explained in news: http://grub.org/?q=/node/241 )?
Our stable server have problem with usernames which contains dots and whitespaces. So, if you want help Grub, you must try our test server. I'm sorry for troubles.

#10

Alas, changing the server had no effect. I'm still unable to upload the arc-file.

#11

Ok, question: did you delete old crawled .arc.gz files? If you switch to new server, you can send to it only new .arc.gz files.

#12

I reinstalled the program, and just now I've uploaded my first file for Grub. Wonderful, thanks for your help..

#13

I think I found the reason,
It appears the grub python client uses the "w" mode to open its file, which is text mode. This translates all \n characters to the native windows \r\n character which then means that count is wrong. Every \n character which has a len of 1 byte becomes a \r\n character which has a len of 2 bytes. This throws the count off and the server thinks the arcfile is invalid.

This reason is only apparant in windows versions as unix and unix like systems use \n as their native newline character so no translation appears.

The solution is to change the mode from "w" to "wb" when opening the arcfile which will then open it in binary mode, so no translation of newline characters occur.

So the line in Worker.run()
changes from:
self.arc = open(self.filename, 'w')
to:
self.arc = open(self.filename, 'wb')

Ducky.

#14

Κατάσταση:active» fixed

Fixed in 0.3

#15

Κατάσταση:fixed» closed

Automatically closed -- issue fixed for two weeks with no activity.