A comparison was performed between the use of graphene oxide (GO) and reduced graphene oxide (rGO) as a hole extraction layer (HEL) in organic photovoltaic (OPV) cells with poly(3-hexylthiophene):phenyl-C61-butyric acid methyl ester. Hydrazine hydrate (HYD) and the thermal method (Thermal) were adopted to change the GO to rGO. The GO HEL was deposited on an indium tin oxide electrode by spin coating, followed by the reduction process to form the rGO HELs. The success of the reduction processes was confirmed by X-ray diffraction, Raman spectroscopy, X-ray photoemission spectroscopy, transmittance, and 2-point probe method. The OPV cell with the GO (-3 nm) HEL exhibits an increased power conversion efficiency (PCE) as high as 2.5% under 100 mW/cm2 illumination under air mass conditions, which is higher than that of the OPV cell without HEL, viz. 1.78%. However, the PCE of the OPV cell with rGO HEL is not high as the values of 1.8% for the HYD-rGO and 1.9% for the Thermal-rGO. The ultraviolet photoemission spectroscopy results showed that the work function of GO was 4.7 eV, but those of HYD-rGO and Thermal-rGO were 4.2 eV and 4.5 eV, respectively. Therefore, it is considered that GO is adequate to extract the holes from the active layer, but HYD-rGO and Thermal-rGO are not appropriate to use as HELs in OPV cells from the viewpoint of the energy alignment.