MPC Status Page: Archive (2003 August)

Recent problems are listed elsewhere. Index of other older problems..

Older Enhancements and Resolved Problems

• Web server

  2003 August 30: 16:00. The webserver machine SCULLY's sole remaining SCSI interface card is reporting controller errors. This will require a service call. Until this is fixed we are reverting to manual extraction and ACKing of incoming observation batches.
  • 17:15. Service has been called. We are waiting to hear from the local field office. SCULLY has been rebooted and is currently serving NEOCP ephemerides. The machine will obviously need to be taken off-line to fit the new interface card. We will probably not be able to give much advance warning of this. Sorry.
  • 20:15. Field service dude arrived 15 minutes ago. Interface card has been replaced. System booted as normal. Normality should be restored.

• Access to Telnet Computer Service

  2003 August 26: 15:00. A number of users have reported that they have been unable to make a connection to the telnet Computer Services. The two machines that run these services are up and running and they respond to telnet connections from observatory-internal machines. Problems have also been reported with access to the anon-ftp server and receipt of designation e-mails. This suggests that some change in the network configuration is preventing telnet connections from being made. We will try and investigate this with the networking group at the observatory.
  • Aug. 27: 15:00. The connectivity problems are being blamed on a recent router "upgrade" that means that default routes are no longer broadcast. We are configuring static routes on our machines.
  • The anon-ftp problems are related to a tightening of the rules regarding anon-ftp passwords. You must give a password that looks like a fully qualified e-mail address (e.g., username@fullyqualified.domain.name, although username@ is apparently allowed). But generic or default email addresses (such as mozilla@, guest@ and anonymous@) will not work. Warnings are returned by the anon-ftp servers if an invalid password is offered.
    • Note that you need to explicitly set your browser's anon-ftp password in order for the ftp links to work. The default values in browsers such as Mozilla and Netscape will NOT work.
  • We are informed that an e-mail blacklist used by popular spam filter software has put cfa.harvard.edu on their blacklist.
  • We believe these problems (with the exception of the e-mail blacklist) are now fixed. If you find otherwise, please send e-mail to the usual address.

• Normal service

  2003 August 18: 14:00. The procedures that failed over the previous two days as a result of the disk space problems on the CF webserver have been rerun. Normality has returned. If you find something still wonky with our site, please let us know (but please check that it is still a problem before reporting it).

• Mirror Pages

  2003 August 18: 11:00. We have set up an index page for the mirrored web pages, as it seems that many users did not know that we had mirror pages!

• CF Webserver disk space problems

  2003 August 18: 10:00. We have now talked to the CF. They have moved some of our web pages to an alternate (larger) disk. We will now rerun those procedures that failed. Some comments:
  • The VMS side of the operation worked flawlessly throughout this crisis.
  • The mirrored pages remained accessible, as did all the scripts served from the VMS webserver.

• CF Webserver disk space problems

  2003 August 17: 22:00. The CF webserver is now down to zero space. This means that, because of the behavior of Unix file systems, any file that is ftp'ed over to the webserver blanks out the existing file (rather than using a temporary file and a transparent rename). We would be examining the entire MPC website to see what can be deleted to clear space, except that the snapshot system means that any space freed up would be claimed by the snapshotted files! At time of writing, we (as an unprivileged user) still have no delete access to the snapshot directories. This is very frustating and requires that tonight's DOU MPEC be abandoned.
  • Attempts to delete files led to no space being freed. Total files deleted amounted to 1.6 GB. Disk usage remained at 100 percent with no free blocks!
  • There is no problem at the VMS end of the operation. All the grief is coming from the outside CF end.
  • Normal service will be restored as soon as possible on Monday morning.
  • We intend more pages to be mirrored on our VMS webserver. This work will proceed as time permits.

• CF Webserver disk space problems

  2003 August 17: 11:00. The CF webserver is still suffering from a lack of available disk space and this caused the update for the latest mid-month MPS batch to fail. The problem stems from the use of sixteen "snapshot" directories which allow access to accidentally-deleted files by giving access to copies of the source directory that were made at hourly/daily/ weekly intervals over the past 8 days or so. Snapshots are a wonderful idea that allow a user to recover a recently deleted file without bothering the Computation Facility, but the individual user has no delete access over the files in the snapshot directories (even though the files are shown as being owned by the user). There are currently many GB of "snapshotted" files (from just one directory) that we know can safely be deleted, but we don't have permission to do so!
  • The CF has suggested that our files be moved to another (larger) device. We are informed that this change will be transparent both to outside users and to our internal scripts that update our pages. We shall investigate this on Monday.
  • If space can be freed up today (it is dependent on a sysadmin being in the Computation Center), the files will be put up asap.

• MPEC 2003-Q01 associated data files

  2003 August 16: 15:00. The MPCUPDATE files associated with last night's DOU MPEC are incomplete due to the CF's webserver running out of disk space. We are fixing the problem by republishing all the orbits from Q01 on the next DOU MPEC. But unless the CF fixes its disk space problem, the problem will repeat. Of course, it is a weekend (see similarly badly timed problem below)...
  • We decided against republishing the Q01 orbits. We have fixed the affected datafiles and added the index page. The on-line MPEC (which was truncated due to the disk space problem) has been restored.
  • The amount of available disk space on the CF webserver has increased from 5 MB to 1.4 GB.

• UPS failure

  2003 August 15: 12:00. The UPS on one of our cluster machines failed last night. The affected machine is still receiving power but has no surge protection and so is now subject to the sometimes iffy power supply in Cambridge. Replacement batteries will be ordered on Monday (our Division Administrator is away this week). There is a slight risk that the machine will suffer a power loss before the batteries arrive. It is probable that the machine will rejoin the cluster without problem. But there is a slight chance that it will mess things up.

• DISCSTATUS

  2003 August 7: 12:00. The DISCSTATUS reports sent out last night should be ignored. The list of objects should be correct, but the orbital information may be incorrect. The program that generates the reports requires datafiles that were generated daily by the old MPES updating routines. These files are no longer updated. The program needs to be modified to use the new data structures. This will be done as soon as possible.

• Replacement of MPEPH3.COM

  2003 August 4: 17:10. When we upgraded the MPES, we omitted to update one web script. This meant that new elements (for both previously-designated and newly-designated objects) were not available through pre-generated pages such as the Dates of Last Observation of NEOs page. This anomaly has been corrected. Some work needs to be undertaken to make all the forms consistent, but the pages should again function.

Index to the CBAT/MPC/ICQ pages. Credits

MPC homepage