MPC Status Page: Archive (2006 July-December)

Recent problems are listed elsewhere. Index of other older problems..

Older Enhancements and Resolved Problems

• MPC Webserver machine

  2006 Dec. 28: 10:53. It appears that the MPC webserver machine is hanging again.
  • 11:15. Bike is at repair shop. Going to pick it up, then head into office.
  • 12:00. Bike not ready. Will not be able to get into office until shortly after 13:00.
  • 13:40. Scully rebooted.

• Cluster problem?

  2006 Dec. 13: 17:20. It appears that the MPC cluster is experiencing a hang. Will head into office after 19:00 (lighter traffic...) to diagnose the problem.
  • 20:00. Cluster hang cleared.

• MPC Webserver machine

  2006 Dec. 9: 17:20. It appears that the MPC webserver machine is hanging again. Cannot head into office to fix problem until late this evening.
  • 22:40. The webserver machine was restarted a few minutes ago.

• Upcoming Network Maintenance (Dec. 5)

  2006 Nov. 29: 12:15. We have just been informed that Harvard will be performing maintenance on one of its core routers on Tuesday, December 5. The work will begin at 05:00 EST and should be completed by 07:00. During this maintenance window there may intermittent losses of connectivity into and out of the observatory.

• MPEC 2006-W86

  2006 Nov. 25: 09:40. Due to a disk problem overnight, the orbits published on MPEC 2006-W86 did not make it into the MPCORB file or the files in the ECS. The orbits will be republished on the next DOU MPEC.
  • 11:40. In order to alleviate some of the immediate disk problems, the residual blocks for the unnumbered minor planets, comets and natural satellites are being moved to another device. While this move is underway, residuals blocks may be unavailable. It may take some time as the transfer is having to be done disk to disk, rather than via tape, as this move is being done remotely.

• CfA Mail Gateway Down Time

  2006 Nov. 24: 14:35. We have just been informed that the CfA mail gateway machine will be shutdown at 17:30 today in order to install "some important OS patches". The outage is expected to last 30 minutes. During this time, incoming mail routed via the mail gateway will not be received, but should queue up on the sending computer for subsequent resend attempts.
  • 17:58. We have been informed that the necessary upgrades have been completed and e-mail delivery has resumed.

• MPEC 2006-W43

  2006 Nov. 21: 15:40. Due to a disk problem overnight, the orbits published on MPEC 2006-W43 did not make it into the MPCORB file or the files in the ECS. The orbits will be republished on the next DOU MPEC.
  • Nov. 22: 16:10. There are still problems with some of the ECS orbit files. These will be corrected with the issuance of the next DOU MPEC (or before if a special MPEC is issued).

• MPC Webserver machine

  2006 Nov. 15: 23:30. It appears that the MPC webserver machine is hanging again. Heading into office to fix the problem.
  • Nov. 16: 00:35. The webserver machine was restarted a few minutes ago.

• Dates of Last Observation of Numbered Minor Planets in MPES

  2006 Nov. 11: 22:43. A problem affecting the display of the date of last observation of the numbered minor planets was identified and fixed.

• MPC Webserver machine

  2006 Nov. 11: 21:30. It appears that the MPC webserver machine is hanging yet again. We are headed into the office to fix the problem. Replacement of the webserver machine seems to the only option.
  • 22:32. The webserver machine was restarted a few minutes ago.

• MPC Webserver machine

  2006 Nov. 8: 23:50. It appears that the MPC webserver machine is hanging again. Since the last attempt to do a remote reboot didn't work, we will have to delay fixing this problem until we are present in the office tomorrow. Web services may be affected.
  • Nov. 9: 10:50. As before, the webserver machine did not respond to a remote reboot request. The machine was rebooted manually. We expect that there will be more problems in the coming days.
  • 10:55. Delivery of incoming e-mail to staff e-mail addresses is unaffected by the problems with the webserver machine. The only problem is that ACKs cannot be sent when the problems are occuring.

• MPC Webserver machine

  2006 Nov. 7: 11:50. It appears that the MPC webserver machine is hanging. We will attempt a remote reboot. If that doesn't work, we'll go and manually reboot the machine. Web services will presumably be off-line until the webserver machine is back on-line.
  • 12:20. The webserver machine did not respond to a remote reboot request. The machine was rebooted manually.

• Missing Comets in MPChecker

  2006 Nov. 6: 13:48. It seems that certain comets are not showing up when they are supposed to in MPChecker. The comets do have elements in the file used by MPChecker, so the problem probably lies in the code that decides whether or not to output an object. Investigations are continuing.
  • Nov. 7: 14:20. It appears that the problem is not in the web script, but in the file of comet elements that was used to prepare the search files immediately after the last MPCs were prepared. This file is being regenerated.
  • 14:53. The missing comets are now being displayed in MPChecker. A full resolution of this problem will occur after the next DOU MPEC.

• Upcoming Network Maintenance (Nov. 7)

  2006 Nov. 1: 11:10. We have been informed that Harvard will be performing maintenance on one of its core routers on Tuesday, November 7. The work will begin at 05:00 EST and should be completed before 06:00. During this maintenance window there may intermittent losses of connectivity into and out of the observatory.

• "Must" vs "Should"

  2006 Oct. 23: 23:15. Following an e-mail query, it is worth remarking that descriptions on our pages described with "should" are recommendations that it is in someone's best interest to follow. Descriptions described with "must" are requirements that must be followed. A number of pages have had occurences of "should" changed to "must" to reflect this difference. Further instances may need changing as they are noted.

• Computer Move on October 7

  2006 Oct. 5: 18:41. On the morning of Sunday, October 7, a number of the cluster machines will be moved physically from their current locations (see Aug. 8 message). This move is expected to take a couple of hours. Internal MPC processing will be suspended during the entire period of the move. Web script service will be unavailable for some portion of the move period. The move is scheduled to begin at 10:00 EDT.
  • 09:55. Arrived in office. Move will begin as soon as I've had my breakfast bar...
  • 11:08. UPS and new networking components have been moved (delay caused by difficulty in removing expired UPS battery). Now to move SCULLY and APOLLO...
  • 13:59. The last of the machines have been moved and the cluster is now fully restored. Before restarting automated processes we want to do some testing, principally of the shadow set which got taken down in an unintended fashion.

• MPChecker

  2006 Oct. 5: 18:40. A new option has been to MPChecker to allow the designations of objects found within specified search areas to be displayed in unpacked or packed format. The default is unpacked format.

• Problems this morning

  2006 Sept. 30: 09:47. Sometime between 02:20 and 05:15 EDT this morning, there was an "event" that caused a slowdown in the cluster. At this stage, we're still trying to track down the nature of this "event" and determine what was happening at that time. The web server on SCULLY was restarted, as the "event" seems to had particular impact on this machine.

• Network outage

  2006 Sept. 23: 11:15. At around 07:25 EDT the CfA network was disconnected unexpectedly from the rest of the Internet. Connectivity was restored around 10:59. No reason has yet been given for the outage. The MPC cluster was unaffected internally by this outage.

• DISCSTATUS

  2006 Sept. 8: 14:40. The most recent DISCSTATUS may be missing discoveries included in the filing of the data from the 2006 Sept. 7 MPCs. This is due to a mistake made when combining the new discovery information with the main discovery archive. This problem has been corrected in the master files. Any missing objects should reappear in the next DISCSTATUS report.

• MPEC 2006-Q39

  2006 Aug. 29: 11:40. The identification 1998 QP = 2006 CY given in the title of MPEC 2006-Q39 is correct. The identification listed in the orbital elements section of that circular (and repeated on the DOU MPEC) is erronous. A correction will appear on the next DOU MPEC. Until this correction appears, the elements for 2006 QY will not be accessible in the MPES.

• Forthcoming network maintenance (August 29)

  2006 Aug. 23: 16:40. We have been informed that Harvard will be performing network maintenance which will cause an interruption to the CfA internet connectivity for about 30 minutes sometime between 04:30 and 07:00 EDT on Tuesday August 29.

• PS and PDF versions of the MPCs, MPSs and MPOs

  2006 Aug. 9: 11:23. It seems that the posting of the PS version of the Aug. MPC batch and the moving of the July files to the "last month" directory failed . Unfortunately, this failure occurred in a part of the update procedure that runs on a machine where we have no logs to examine to see what the problem was. The files will have to be replaced manually.
  • 12:12. The files have been replaced.

• Physical Relocation of Computers

  2006 Aug. 8: 21:53. We have been informed that some of our cluster machines need to be moved in order to comply with CfA policy on computer placement. This relocation will take place in two phases over the next week or so. In the first phase, one machine will be moved to the new location, to see whether it can "see" the other cluster members and, assuming that it can, whether it can operate . In the second phase, six other machines will be moved. The first phase will probably not affect MPC operation or the website. The second phase will, as one of the machines being moved is the MPC webserver. Only limited advance notice of this move will be possible.
  • Aug. 9: 17:09. It seems that the first phase will be undertaken on Thursday, Aug. 10.

• CF Mail Gateway Down on July 31

  2006 July 26: 11:02. We have been informed that the CF mail gateway will be shutdown at 07:30 EDT on Monday July 31 in order to install important OS patches. The downtime is expected to last 30 minutes. Incoming mail will not be received during the outage, but the mail should queue up on the sending computer for subsequent resend attempts.

• Massive DAILY.DAT files

  2006 July 15: 12:40. In past months, the DAILY.DAT file posted on the MPCORB ftp site has occasionally been massive, as it appears to be a copy of the MPCORB.DAT file. At the same time, MPCORB.DAT files would be suspiciously small. We now think this happens only when two sftp transfers happen at the same time and that it is a consquence of the convulted way multiple local files are merged in to one remote file, a process made necessary by the lack of "append" in sftp. A fix has been put in place to hopefully alleviate this problem in the future and last night's DAILY.DAT has been replaced with the proper version.

• Forthcoming network maintenance (July 20)

  2006 July 13: 15:40. We have been informed that Harvard will be performing network maintenance which will cause an interruption to the CfA internet connectivity between 05:00 and 06:00 EDT on Thursday July 20.

• Machine problem

  2006 July 3: 12:00. One of our machines has developed a problem with its SCSI interface, with the upshot that a few directories are unavailable to the cluster. These "missing" directories do not affect our internal operations (i.e., we can receive, process and file observations as normal), but DOU MPEC preparation is not possible. One of the unavailable directories holds some of the template files needed for some of the web services (such as the MPChecker) and a special file needed when requesting numbered minor planets from the MPES: affected services may be partially or completely unavailable.
  • July 5: 11:30. The 4th July holiday is delaying shipment of the replacement SCSI interface (it is being sent Overnight Fed-Ex): it should be here tomorrow. Normal external service should then resume.
  • July 5: 16:20. When the interface is replaced, the directory required by the web services will be copied to the shadow set so that it remains available if this problem repeats in the future.
  • July 6: 09:30. The SCSI interface has been delivered to the mail room. No idea if it'll arrive up here with first internal mail delivery...
  • July 6: 14:30. SCSI interface still hasn't arrived up here. Mail room claims it was delivered around 10:30...
  • July 6: 15:30. SCSI interface finally arrives in office. Can't remove old interface card... Eventually gave up and installed new card in another PCI slot. Affected machine rebooted, couple of directories moved onto shadow set, web services (apparently) back to normal.

Index to the CBAT/MPC/ICQ pages. Credits

MPC homepage