Help → No response from POPfile

No response from POPfile

I link to several e-mail accounts on different servers using Pegasus Mail and POPfile on Windows XP SP2. Error messages from Pegasus relating to any of these accounts have been appearing intermittently when I try to download mail:

[*] Connection established to 127.0.0.1

0040 +OK POP3 POPFile (v1.0.0) server ready
0035 USER pop.1and1.co.uk:xxxxxxxxxx [redacted]
0035 -ERR no response from mail server

If I close the error message and try to view the POPfile UI in a browser, it hangs. If I shut down POPfile and restart it, the error (usually) does not reappear.

Can anyone suggest a cause and solution?

  • Message #200

    Try using the Message Capture utility to run POPFile. To do this, shutdown POPFile then use the following shortcut to run POPFile:

    Start -- All Programs -- POPFile -- Support -- Message Capture utility

    Normally POPFile's console messages are hidden. This shortcut will display them in a scrollable window. In addition to some startup messages there may be some warnings or error messages. The utility's report may shed some light upon what is going wrong.

    Brian

    • Message #201

      Try using the Message Capture utility to run POPFile. To do this, shutdown POPFile then use the following shortcut to run POPFile:

      Start -- All Programs -- POPFile -- Support -- Message Capture utility

      I've tried that. The startup messages are


      POPFile Message Capture Utility v0.1.8


      POPFILE_ROOT = C:\PROGRA~1\POPFile
      POPFILE_USER = D:\Internet\POPFile
      Using 'popfileif.exe' to run POPFile


      (report started 21-Jun-2008 @ 20:43:31)


      POPFile Engine loading

      Loading...

      {core: windows}
      {core: config history logger mq}
      {classifier: bayes wordmangle}
      {interface: html xmlrpc}
      {proxy: nntp pop3 smtp}
      {services: imap}

      POPFile Engine v1.0.1 starting

      Initializing...

      {core: config history logger mq windows}
      {classifier: bayes wordmangle}
      {interface: html xmlrpc}
      {proxy: nntp pop3 smtp}
      {services: imap}

      Starting...

      {core: config history logger mq windows}
      {classifier: bayes wordmangle}
      {interface: html}
      {proxy: pop3}
      {services:}

      POPFile Engine v1.0.1 running

      When the fault occurs, no additional message appears.

      • Message #202

        Thanks for the report. As you say, it does not show any unexpected messages.

        The problem might simply be something to do with the volume of mail varying between the various accounts. By default the Windows version of POPFile only processes one email account at a time, so if Pegasus uses POPFile to checks accounts A, B, C, D, and E then POPFile will process all of the new mail for account A before it moves on to check all of the new mail for account B, then all of the new mail for account C, and so on.

        Therefore it is possible that sometimes Pegasus will timeout and complain about a lack of response - even though nothing has gone wrong.

        You might be able to stop this from happening if you increase the timeout Pegasus uses to detect this "lack of response" problem.

        Here is some information from the Release Notes that explains this phenomenon:

        <snip>

        2. ON WINDOWS I WANT TO CHECK MULTIPLE EMAIL ACCOUNTS SIMULTANEOUSLY.

        Because the time taken to start a new process on Windows is long under Perl there is an optimization for Windows that is present by default: when a new connection is made between your email program and POPFile, POPFile handles it in the 'parent' process. This means the connect happens fast and mail starts downloading very quickly, but it means that you can only download messages from one server at a time (up to 6 other connections will be queued up and dealt with in the order they arrive) and the UI is unavailable while downloading email.

        You can turn this behavior off (and get simultaneous UI/email access and as many email connections as you like) on the Configuration panel in the UI by making sure that "Allow concurrent POP3 connections:" is Yes, or by specifying --set pop3_force_fork=1 on the command line.

        The default behaviour (no concurrent POP3 connections) can cause email clients to time out if several accounts are being checked (because POPFile only handles one account at a time it can take a while to process all of the accounts).

        If SSL support is being used then the default setting (no concurrent POP3 connections) _MUST_ be used otherwise POPFile crashes.

        </snip>

        Brian

        • Message #203

          The problem might simply be something to do with the volume of mail varying between the various accounts. By default the Windows version of POPFile only processes one email account at a time, so if Pegasus uses POPFile to checks accounts A, B, C, D, and E then POPFile will process all of the new mail for account A before it moves on to check all of the new mail for account B, then all of the new mail for account C, and so on.

          I'd considered that possibility and set Pegasus to only download one account's mail at a time. It made no difference. If I close POPfile and restart it, the problem goes away, at least for that attempt to get mail. I've also noticed - this is a very subjective observation! - that the error seems to happen more often when there is either no mail or only one or two messages waiting. If there are (say) a dozen, it seems to work better.

          • Message #204

            The problem might be something to do with the content of a particular message.

            By default POPFile's log file does not contain much detail (logger_level is set to 0 by default). If you increase the logging level to 2 then the log file will include the entire text of every message that POPFile processes, so we can then see if it goes wrong in the middle of a message.

            Be warned that at level 2 the log file can grow very large, especially if you get a lot of mail. Another thing that can make the log file large is if you use the UI a lot.

            The entries in the log file are all timestamped to help you navigate through the file and each time POPFile starts up and shuts down is also recorded in the log file.

            Some wiki references:

            Preparing a log file

            What is being logged in POPFile*.log

            Brian

            • Message #205

              By default POPFile's log file does not contain much detail (logger_level is set to 0 by default). If you increase the logging level to 2 then the log file will include the entire text of every message that POPFile processes, so we can then see if it goes wrong in the middle of a message.

              This is what I get when the fault occurs:

              > 2008/6/22 12:36:16 1444: pop3: 301: +OK POP3 POPFile (v1.0.1) server ready[0d][0a]
              > 2008/6/22 12:36:16 1444: pop3: 194: Regexps: ^USER XXXXXX
              > 2008/6/22 12:36:16 1444: pop3: 209: Command: --USER XXXXXX--
              > 2008/6/22 12:36:16 1444: mq: 377: post LOGIN (m37749005-qi)
              > 2008/6/22 12:36:16 1444: mq: 384: queuing post LOGIN (m37749005-qi)
              > 2008/6/22 12:36:16 1444: mq: 386: LOGIN queue length now 0
              > 2008/6/22 12:36:16 1444: pop3: 512: Attempting to connect to POP server at pop.1and1.co.uk:110
              > 2008/6/22 12:36:16 1444: pop3: 521: Connected to pop.1and1.co.uk:110 timeout 60
              > 2008/6/22 12:36:16 1444: pop3: 559: Connection returned: 
              > 2008/6/22 12:36:16 1444: pop3: 295: auth plaintext
              > 2008/6/22 12:36:16 1444: pop3: 301: USER XXXXXX
              > 2008/6/22 12:36:16 1444: pop3: 301: -ERR no response from mail server[0d][0a]
              > 2008/6/22 12:36:16 1444: mq: 377: post CMPLT (1444)
              > 2008/6/22 12:36:16 1444: mq: 388: dropping post CMPLT (1444)
              > 2008/6/22 12:36:16 1444: pop3: 668: POP3 proxy done
              > 2008/6/22 12:36:17 1444: mq: 128: Message LOGIN (m37749005-qi) ready for delivery
              > 2008/6/22 12:36:17 1444: mq: 131: Delivering message LOGIN (m37749005-qi) to html
              

              This implies that the fault lies with 1and1, does it not? This is odd, as direct connection to their mail servers doesn't produce the error.

              • Message #206

                This implies that the fault lies with 1and1, does it not?

                I am not sure. Manni and Naoki know much more about this side of things than I do.

                12:36:16 1444: pop3: 512: Attempting to connect to POP server at pop.1and1.co.uk:110
                12:36:16 1444: pop3: 521: Connected to pop.1and1.co.uk:110 timeout 60
                12:36:16 1444: pop3: 559: Connection returned:
                12:36:16 1444: pop3: 295: auth plaintext
                12:36:16 1444: pop3: 301: USER XXXXXX
                12:36:16 1444: pop3: 301: -ERR no response from mail server[0d][0a]
                12:36:16 1444: mq: 377: post CMPLT (1444)

                One thing I noticed is that the "no response from mail server" log entry has the same timestamp as the login attempt so it does not look like POPFile waited very long to see if the mail server responded. But as I said, I know almost nothing about how this part of POPFile works.

                Brian

                • Message #207

                  But as I said, I know almost nothing about how this part of POPFile works.

                  Almost as soon as I posted that reply I remembered that a few days ago one of the mail servers I use was out-of-action for a few days so I looked back through my old POPFile logs to see how POPFile handled that situation.

                  Here is what I found when the mail server was down:

                  01:05:00 -4212: pop3: 512: Attempting to connect to POP server at pop3.myrealbox.com:110
                  01:05:00 -4212: pop3: 521: Connected to pop3.myrealbox.com:110 timeout 60
                  01:05:21 -4212: pop3: 559: Connection returned: -ERR AVG POP3 Proxy Server: Cannot connect to the mail server![0d][0a]
                  01:05:21 -4212: pop3: 301: USER XXXXXX
                  01:05:21 -4212: pop3: 301: -ERR no response from mail server[0d][0a]
                  01:05:21 -4212: pop3: 668: POP3 proxy done
                  

                  and here is a successful attempt at checking for new mail on that server:

                  17:23:29 -2420: pop3: 512: Attempting to connect to POP server at pop3.myrealbox.com:110
                  17:23:29 -2420: pop3: 521: Connected to pop3.myrealbox.com:110 timeout 60
                  17:23:29 -2420: pop3: 559: Connection returned: +OK AVG POP3 Proxy Server 7.5.510/7.5.524 [270.4.0/1507][0d][0a]
                  17:23:30 -2420: pop3: 301: USER XXXXXX
                  17:23:30 -2420: pop3: 301: +OK Password required[0d][0a]
                  17:23:30 -2420: pop3: 301: PASS XXXXXX
                  17:23:30 -2420: pop3: 301: +OK[0d][0a]
                  17:23:30 -2420: pop3: 301: STAT[0d][0a]
                  17:23:30 -2420: pop3: 301: +OK 0 0[0d][0a]
                  17:23:30 -2420: pop3: 301: QUIT[0d][0a]
                  17:23:30 -2420: pop3: 301: +OK myrealbox.com signing off[0d][0a]
                  17:23:30 -2420: pop3: 668: POP3 proxy done
                  

                  My system is configured like this:

                  Thunderbird -- POPFile -- AVG Personal Email Scanner -- Internet -- Mail Server

                  which is why the "Connection returned" log entry mentions AVG.

                  Looking again at your log extract it seems that 1and1's mail server should have responded to the initial connection attempt (i.e. the "Connection returned:" entry in the log should have had some text after the colon) and it should also have responded to the "USER" command.

                  Brian

  • Message #208

    Brian already pointed you in the right direction.

    The error message "-ERR no response from mail server" is not generated by Pegasus, but by POPFile. POPFile will generate this error in (at least) two situations:

    1. The connection to the server timed out. But that doesn't seem to be the case here (see the timestamps in the log file).

    2. POPFile couldn't connect to the server. I guess this must be the problem here. Either your connection is flaky or the POP3 server is not always responding to connection attempts.

    Is it always the same server that triggers the error? Maybe the server doesn't like to be contacted too often. How often do you check for mail?

    Of course, this still leaves us with the question why POPFile isn't responding at all after this happens. This must be some kind of bug in POPFile.

    • Message #209

      Is it always the same server that triggers the error? Maybe the server doesn't like to be contacted too often. How often do you check for mail?

      There are two accounts on one server, a third on a different server at the same ISP and a fourth on a completely different system at my cable provider. All four exhibit the same problem (which is partly why I discounted server error, but mainly because the problem doesn't occur if I go direct).

      My connection may be bad in some way (I did wonder about the e-mail protection of Kapersky anti-virus, but turning that off makes no difference). I'm on a very fast cable connection (2Mb/sec) which is usually solid. The only oddity I can think of is that sometimes Web page loads don't happen (blank screen) until I refresh the page. I check for mail no more than about every half hour even at busy times.

      Of course, this still leaves us with the question why POPFile isn't responding at all after this happens. This must be some kind of bug in POPFile.

      Are we then at a dead end?

      • Message #210

        Are we then at a dead end?

        Not yet.

        Perhaps the problem is that the POP3 server is taking a while to respond. Apparently, it doesn't send any greeting:

        12:36:16 1444: pop3: 559: Connection returned:

        We should expect to see the server's banner after that colon.

        You could simply try to raise your timeout in POPFile to some larger value and see whether this makes the problem go away. Go to POPFile's configuration tab and find the timeout widget on the lower left. Double the current value and test.

        Manni

      • Message #211

        Are we then at a dead end?

        No, I don't think so.

        If you have POPFile configured to use the default POP3 listen port (110) I suggest you change this to, say, port 123 and change Pegasus to use port 123 for the accounts which are handled by POPFile. So your setup will then be like this:

        Pegaus -- port 123 -- POPFile -- port 110 -- mail server

        There is more information about this in the wiki (e.g. Proxy Chaining). That wiki page mentions more than just changing the POP3 listen port but I think only that setting needs to be changed.

        In theory Pegasus should be able to use port 110 to communicate with POPFile but I've never trusted that sort of setup and have always used a non-standard POP3 listen port.

        Another thing to look at is what the POPFile log file shows when Pegasus is able to collect mail via POPFile - is there a text string shown after the "Connection returned:" part of the relevant log entry?

        Brian

        • Message #212

          If you have POPFile configured to use the default POP3 listen port (110) I suggest you change this to, say, port 123 and change Pegasus to use port 123 for the accounts which are handled by POPFile.

          Changing the port has cured the problem. Many thanks to you and Manni for your help in pursuing this one to a satisfactory conclusion!

          • Message #213

            Changing the port has cured the problem.

            Glad to hear that this fixed things for you.

            Don't forget to reset the logger_level back to 0 or 1 now (instead of level 2).

            I use level 1 because it gives a lot of information about the POP3 connection handshaking (but does not include any message text or any UI-related log entries - this makes the log file easier to read).

            Brian