The purpose of this study was to develop a new algorithm to detect the inner and outer borders of the retinal nerve fiber layer (RNFL) using optical coherence tomography (OCT). A program featuring a new algorithm was developed using a commercially available software development environment. The algorithm searches for peaks on each sampling line instead of applying conventional thresholding techniques. All circular peripapillary OCT images obtained at the New York Eye and Ear Infirmary from October 1996 to December 2000 were analyzed using the new and the commercially available algorithms. Four hundred one images of 98 patients were analyzed. The detection error rate (defined as number of scans with five consecutive sampling lines with disrupted RNFL borders) was significantly lower with the new algorithm (25/401 images, 6.2%) than with the commercially available algorithm (70/401 images, 17.5%) (P=0.018, chi-square). The new algorithm improved the ability of OCT to detect the borders of the RNFL.