MPC Status Page: Archive (2007 January-June)

Recent problems are listed elsewhere. Index of other older problems..

Older Enhancements and Resolved Problems

• Special-epoch elements for NEAs

  2007 June 29: 10:35. A number of NEAs are in the special daily-epoch elements files under both their numbered and unnumbered designations. A problem that prevented the removal of certain old sets of elements has been identified and fixed. The files are being regenerated.

• Numbered elements in MPES

  2007 June 24: 16:35. A problem that caused the elements of numbered minor planets to be unavailable in the MPES since noon was detected and fixed.

• CF mail server outage (June 20)

  2007 June 15: 15:55. We have been informed that the CF's SMTP server will be rebooted at 08:00 EDT on Wednesday, June 20. During the expected 30-minute outage, incoming mail (e.g., to mpc@cfa) will not be received, but the mail should be resent by the remote systems when the SMTP server is restarted.

• Anonymous ftp server downtime (June 1)

  2007 May 29: 12:35. We have been informed that the CF's anonymous ftp server will be offline for approximately 30 minutes starting at 09:00 EDT on June 1. Access to files stored there will obviously not be possible during this period.

• Magnitudes in MPES and MP/NEOChecker

  2007 May 19: 22:00. A user reported a difference in the predicted magnitude of a numbered object as displayed in the MPES and in the MP/NEOChecker services. A fix for this discrepancy has been put in place.

• MPC Cluster problem

  2007 May 7: 10:30. It seems that one of the machines in the cluster rebooted itself earlier this morning and did not come back up. This machine stores the quick-access indexed observation and orbit files used internally and by some of the web services. The machine will be rebooted as soon as possible.
  • 11:30. The affected machine has been restarted.

• Date of Last Observation of numbered objects in MPES

  2007 May 5: 17:47. A problem with the generation of a datafile used by the MPES is causing the dates of last observation of certain newly-numbered objects not to be displayed by the MPES. The MPES did not expect this situation and would stack dump if an affected object was selected. The code has been tighten to prevent the stack dump and output a meaningful informational message. The problem datafile has been regenerated, recent observations will be reflected in this file following the next DOU MPEC.

• Sky Coverage plots

  2007 May 5: 08:30. A problem related to both the retirement of the old webserver and incorrect directory access permissions on the new webserver caused the script that updated the main Sky Coverage page to not copy newly-generated versions to the webserver. This has been fixed.

• Sluggish response of webserver overnight

  2007 Apr. 23: 08:15. Two separate events overnight caused the response of the webserver to be extremely sluggish (to the point where requests would timeout). The first was a result of a disk being found to be missing to be missing and its reinsertion. The second was a runaway cgi script that clogged up the webserver. We are checking to see why the process that kills runaway cgi scripts was not triggered.
  • 13:00. The process that kills runaway cgi scripts was restarted. The execution batch queue on which it ran originally disappeared when the old webserver was shutdown. It has been restarted on a execution batch queue on the new webserver.

• Moving of MPC/CBAT/ICQ webpages II

  2007 Apr. 19: 12:30. The transfer of the MPC/CBAT/ICQ webpages to a new location will begin at 14:00 today. During the move, our pages should remain available, but certain files may revert to older copies when the new alias is activated. We do not expect to update any web pages during the move. Cgi scripts served from the VMS webserver will continue to be available as normal.
  • 18:30. The webpages have been moved and our internal procedures updated.

• MPC Cluster problems

  2007 Apr. 14: 23:20. It seems that one of the machines in the cluster rebooted itself earlier this evening and has not come up cleanly. For reasons that are not yet understood, this is causing problems across the cluster, including the webserver. We are investigating...
  • 23:36. It seems the problem is that the quick-access indexed observation and orbit files are stored on the machine that has not rebooted. The unavailability of the latter files does impact extraction of orbits and generation of ephemerides in the webserver.
  • Apr. 15: 00:30. We have abandoned tonight's DOU MPEC, postponed the mid-month MPS batch and paused the automated processing procedures. The machine will be rebooted in the late morning.
  • 10:30. The affected machine has been rebooted. Normalcy should be returning.

• Upcoming Network Maintenance (Apr. 29)

  2007 Apr. 13: 14:15. We have been informed that Harvard will be performing hardware upgrades on one of its gateway machines on Sunday, April 29. The work will begin at 01:00 EDT and should be completed by 07:00. During this maintenance window there will be a loss of connectivity into and out of the observatory.

• Moving of MPC/CBAT/ICQ webpages

  2007 Apr. 13: 10:40. We have been informed that the Computation Facility needs to move the MPC/CBAT/ICQ webpages to a new location. This move will take place sometime in the week April 16-20. During the move, our pages should remain available, but certain files may revert to older copies when the new alias is activated. Cgi scripts served from the VMS webserver will continue to be available as normal.

• "File locked" messages from web scripts

  2007 Apr. 6: 14:10. To try and remove the "File locked" messages some users have reported (a problem we've been unable to reproduce internally) a small change to a webserver configuration file has been made and the webserver has been restarted. Use the feedback form to report any occurrences of the "File locked" message.

• MPC Webserver machine

  2007 Apr. 4: 12:30. We are shifting the MPC webserver on to a different machine. When the new webserver is set up, we will shutdown the current webserver. The name "scully" will be retained as an alias for the new webserver, so that no changes to web forms are necessary. There will be a brief period sometime in the next few days when we disable the current webserver (including AUTOACK) to allow us to do a clean copy of data to the new webserver. This may be done with little warning.
  • Apr. 5: 11:38. The cgi scripts that run on the current webserver, as well as the AUTOACK service, are being paused to allow the new webserver to be configured. Copying of the data from current to new webserver, followed by testing of new webserver, may take a few hours.
  • 14:39. The files have been shifted, access to the webserver has been enabled on the network, but the required aliasing of scully.cfa.harvard.edu to the new machine has not yet been made. Access to the cgi scripts will be possible after this aliasing has occurred. AUTOACK should be working normally.
  • 15:57. Correction: AUTOACK isn't working as the new machine is not currently allowed to receive SMTP mail. We have been informed that the request for the alias has been made, but it may be tomorrow morning before the change is made.
  • 17:00. The alias is now visible externally, but not internally.
  • 17:30. The alias is now visible internally.
  • 18:09. The new machine can now receive SMTP mail, but the changes needed to allow the mpc@cfa alias to forward mail to the correct machine do not seem to have been made (i.e., delivery is still going to Scully). Until this is fixed we will forward observation batches manually to the AUTOACK routine.
  • 20:27. From external sites, the web services appear to be accessible again. AUTOACK also seems to be working.

• MPC Webserver machine

  2007 Apr. 1: 11:00. It appears that the MPC webserver machine hung again around 21:06 last night. The machine will be rebooted as soon as possible.
  • 14:57. Webserver has been rebooted.
  • 20:00. Webserver has hung again.
  • Apr. 2: 09:52. Webserver rebooted.
  • Apr. 3: 21:40. Webserver has hung again.
  • Apr. 4: 10:35. Webserver rebooted.

• cgi scripts on CF machines

  2007 Mar. 30: 12:30. Effective immediately, all cgi scripts used by MPC/CBAT that run on CF machines are disabled. This includes the "display IAUC" script and the "show all designations" option in the MPES. Replacements will be worked on, but it may take some time to replace them all.

• MPC Webserver machine

  2007 Mar. 30: 11:22. It appears that the MPC webserver machine hung again around 00:26. The machine is being rebooted.

• MPC Webserver machine

  2007 Mar. 29: 17:48. It appears that the MPC webserver machine hung again around 15:30. The machine has been rebooted.

• MPES bug fixes

  2007 Mar. 29: 00.15. A user reported some non-documentation compliant behavior in the MPES. One problem has been fixed: the non-acceptance of valid dates such as "2007 March 28". The range of allowable east longitudes was extended to the range +/- 360 degrees, rather than the previously allowed 0-360 degrees.

• New Feedback Form

  2007 Mar. 23: 15:20. A new version of the feedback form is now on-line. Pages that reference the old version are gradually being changed to use the new version. At some point in the near future the old version will be removed.

• CMTChecker and NEOChecker

  2007 Mar. 14: 16:00. Two new services that use the MPChecker script to check for just comets and NEOs are now on-line.

• MPC Webserver machine

  2007 Mar. 14: 13:55. It appears that the MPC webserver machine hung again around 11:20. It is being rebooted.
  • 14:03. Turns out not to be a problem with SCULLY. The AUTOACK procedure log file had reached its maximum version number. A fix will be put in place to ensure this doesn't happen again.

• DNS Issues

  2007 Mar. 8: 18:13. It seems that there have been intermittent failures of DNS over the past 24-36 hours at the CfA. This impacted issuance of a CBET last night and the issuance of an MPEC this morning. Both circulars have been sent out manually. We have received no word on whether this is going to continue.

• MPC Webserver machine

  2007 Mar. 8: 13:59. It appears that the MPC webserver machine hung again around 10:43. It is being rebooted.
  • 14:16. The webserver machine has been rebooted.

• CfA Mail Gateway Down Time

  2007 Mar. 6: 11:40. We have just been informed that the CfA mail gateway machine will be shutdown at 06:00 EST tomorrow (March 7) in order to install "some important OS patches". The outage is expected to last 40-60 minutes. During this time, incoming mail routed via the mail gateway will not be received, but should queue up on the sending computer for subsequent resend attempts once the gateway is back on-line.

• MPC Webserver machine

  2007 Mar. 6: 10:03. It appears that the MPC webserver machine hung again around 05:45. The machine has been rebooted.

• MPC Webserver machine and tonight's DOU MPEC

  2007 Feb. 28: 23:25. It appears that the MPC webserver machine is hanging again. The machine will be checked out tomorrow morning. We are hoping that the machine will reboot itself, so that we get a crash dump log to help diagnose where the problem lies. Tonight's DOU MPEC is being canceled.
  • Mar. 1: 10:05. The webserver machine didn't reboot itself, so it has been rebooted manually.
  • 18:05. The webserver machine was taken down for a few moments to check a hardware item at the request of the service organization. The machine has been restarted.

• OS patch installation

  2007 Feb. 27: 14:36. We are planning on installing a number of OS patches (none security related) on the cluster machines starting tomorrow. Each machine will be patched and rebooted at a time that is most convenient for that machine. When the webserver machine is patched, web services will be off-line for the duration of the installation (probably under 30 mins, assuming no surprises). All the machines should be patched by March 4.
  • 16:48. The webserver machine is currently being patched. It will be rebooted in a few minutes. Intermittent web problems may occur while other machines are being patched over the next few days.
  • Feb. 28: 12:29. The patching of the cluster machines has been completed.

• MPC Webserver machine

  2007 Feb. 12: 21:25. It appears that the MPC webserver machine is hanging again.
  • Feb. 13: 10:09. Webserver machine restarted. Restart of ACK procedure complicated by submission of e-mail not formatted properly: this caused multiple ACKs to be sent out. Normalcy has resumed.

• MPC Webserver machine

  2007 Feb. 11: 19:48. It appears that the MPC webserver machine is hanging again.
  • Feb. 12: 08:41. Heading in shortly to fix problem.
  • 10:49. Webserver machine rebooted. Backlog of e-mail will be processed as soon as CF mail node retries delivery.

• Upcoming Network Maintenance (Feb. 6)

  2007 Jan. 29: 15:01. We have just been informed that Harvard will be performing maintenance on another of its core routers on Tuesday, February 6. The work will begin at 04:00 EST and should be completed by 06:00. During this maintenance window there may be intermittent losses of connectivity into and out of the observatory.

• Upcoming Network Maintenance (Jan. 30)

  2007 Jan. 29: 14:43. We have just been informed that Harvard will be performing maintenance on one of its core routers on Tuesday, January 30. The work will begin at 04:30 EST and should be completed by 06:00. During this maintenance window there may intermittent losses of connectivity into and out of the observatory.

• MPC Webserver machine

  2007 Jan. 26: 09:20. It appears that the MPC webserver machine is hanging again. A system configuration modification put in place after the last hang, which we hoped would solve the hanging problem, doesn't seem to be working. Heading into office shortly to fix problem.
  • 11:00. The webserver machine was rebooted.
  • 17:30. The webserver machine was rebooted again.

• Lists of NEOs/TNOs/etc.

  2007 Jan. 25: 18:03. The lists of unusual objects of various kinds are off-line until a new version of the program that prepares said lists is put on-line.

• MPC Webserver machine and tonight's DOU MPEC

  2007 Jan. 21: 15:03. It appears that the MPC webserver machine is hanging again. Heading into office shortly to fix problem.
  • Jan. 21: 16:00. Scully has been rebooted.
  • Jan. 21: 22:40. Hanging again. Reboot has been postponed until tomorrow morning, as there is no guarantee that a fix now will last until the morning. Tonight's DOU MPEC has been abandoned.
  • Jan. 22: 09:52. Scully apparently rebooted itself around 03:20. Queues were just restarted.

• MPC Webserver machine

  2007 Jan. 13: 23:03. It appears that the MPC webserver machine is hanging again. Heading into office to fix problem.
  • Jan. 14: 00:03. Scully was rebooted.

• MPCORB datafile

  2007 Jan. 7: 09:48. A large number of elements are missing from the MPCORB files on the ftp site. Examination of the overnight DOU MPEC logfile shows errors on the ftp server side: apparently the disk used to store the directory /pool (used as an intermediate storgae location while transferring and appending files) filled up. The next update of MPCORB should contain all the elements as normal.

Index to the CBAT/MPC/ICQ pages. Credits

MPC homepage