OPINION OF THE COURT
JAMES HUNTER, III, Circuit Judge:
Mary , Ann Kelly petitions this court to review a final decision of the Railroad Retirement Board.1 The Board denied her application for a disabled child’s annuity under the Railroad Retirement Act, 45 U.S.C. § 231a(d)(l)(iii)(C) (1976).2 Kelly was not found to be disabled. 45 U.S.C. § 231a(d)(3) (1976).3 This court has jurisdiction to review the Board’s decision. 45 U.S.C. § 231g (1976).4 Kelly presses two grounds for reversal: (1) that the agency violated her 5th amendment right to due process in that it committed serious procedural errors in its internal appeals procedures; (2) that the agency decision, finding her not under a disability, is not supported by substantial evidence on the record. We agree with both of Kelly’s contentions. The decision of the Board will be reversed.
I.
Kelly applied for a disabled child’s annuity under the Railroad Retirement Act. 45 U.S.C. ,§ 231a(d)(l)(iii)(C) (1976). She filed her application more than a month before her twenty-second birthday. To be eligible for a disabled child’s annuity, the applicant, irrespective of age at time of application, must suffer from a disability which began before the age of twenty-two. Kelly claimed that she was under a mental disability, severe depression.
At the administrative hearing Kelly testified that she began to suffer from anxiety, insomnia and depression during high school. Unable to sleep for several days on end, she could not concentrate on her studies. Catching up on lost sleep caused her to miss school frequently. Her insomnia interrupted her class attendance to the extent that she was denied a diploma at graduation because she failed to meet the school attendance requirement.
[489]*489After high school, Kelly enrolled in college, but continued to suffer from the same symptoms. She consulted the psychiatrist in residence at the college who prescribed medication, an antidepressant. Its use produced no improvement. On the advice of the school psychiatrist, Kelly withdrew from school. Because she terminated her student status, she was ineligible to continue treatment under the school psychiatrist’s care. Kelly then sought professional treatment at the Rutgers Community Mental Health Center as an indigent. At Rutgers she consulted four psychiatrists and. one psychologist, each of whom submitted a report to the Railroad Retirement Board evaluating Kelly’s mental condition.
Meanwhile, Kelly obtained a series of clerical jobs. She lasted a few days at each. The same symptoms continued: her insomnia prevented punctual appearance at work. She complained that anxiety, produced by the presence of co-workers, interfered with her task performance.
On June 10, 1975, Kelly applied to the Railroad Retirement Board for a child’s disability annuity, based on the earnings of her father, a deceased railroad worker. 45 U.S.C. § 231a(d)(l)(iii)(C) (1976). For the purpose of the Act, a child is under disability if his permanent physical or mental condition prevents him from engaging in any regular employment. Id. at § 231a(d)(3).5
In March 1979, more than three years, nine months after Kelly filed her application, the Railroad Retirement Board issued its final decision finding her ineligible. The Board found that “it has not been established that Appellant is unable to engage in any regular employment by a disability which began before the age 22.”
This court may “enter a decree affirming, modifying, or reversing the decision of the Board, with or without remanding the cause for rehearing.” 45 U.S.C. § 355(f) (1976).6 Appellant urges reversal of the Board decision, challenging the sufficiency of the procedure and substantability of the evidence on which the decision is based.
II.
Kelly urges that the administrative procedure that determined her ineligible to receive a child’s disability annuity failed to comply with due process. The threshold question is whether a claimant initially denied benefits has a sufficient property interest in the benefits to be protected by the fifth amendment guarantee of Due Process. See generally, Board of Regents v. Roth, 408 U.S. 564, 576-79, 92 S.Ct. 2701, 2708-2710, 33 L.Ed.2d 548 (1972); Perry v. Sindermann, 408 U.S. 593, 599-603, 92 S.Ct. 2694, 2698-2700, 33 L.Ed.2d 570 (1972). We conclude that he does.
“To have a property interest in a benefit, a person . . . must . . . have a legitimate claim of entitlement to it.” Board of Regents v. Roth, supra, 408 U.S. at 577, 92 S.Ct. at 2709. “Such benefits are a matter of statutory entitlement for persons qualified to receive them.” Goldberg v. Kelly, 397 U.S. 254, 262, 90 S.Ct. 1011, 1017, 25 L.Ed.2d 287 (1970) (discussing welfare benefits). The welfare claimants in Goldberg “had a claim of entitlement to welfare payments that was grounded in the statute defining eligibility for them. The recipients had not yet shown that they were, in fact, within the statutory terms of eligibility.” Board of Regents v. Roth, supra, 408 U.S. at 577, 92 S.Ct. at 2709. Nevertheless, the Supreme Court required the administrative process determining eligibility to comply with due process. Goldberg v. Kelly, supra, 397 U.S. at 260-266, 90 5. Ct. at 1016-1019; Board of Regents v. Roth, supra, 408 U.S. at 577, 92 S.Ct. at 2709 (discussing Goldberg). See also, Schware v. Board of Bar Examiners, 353 U.S. 232, 238-239, 77 S.Ct. 752, 755-756, 1 L.Ed.2d 796 (1957) (due process applies to eligibility for professional employment.) [490]*490As due process applied to the determination of the eligibility of the welfare claimants in Goldberg v. Kelly, so it applies to the determination of Mary Ann Kelly’s eligibility for a child’s disability annuity.
We note that in Goldberg v. Kelly, the posture of the welfare claimants was that they had been determined ineligible to continue receiving benefits. 397 U.S. at 256, 90 S.Ct. at 1014. Accord, Escalera v. New York City Housing Authority, 425 F.2d 853 (2d Cir. 1970) (public housing). In this case, Kelly’s ineligibility was determined at the outset, and she has never received a disabled child’s annuity. Nevertheless, due process must attach to the process of determining ineligibility, whether at the outset or after receipt of benefits. Wright v. Cali-fano, 587 F.2d 345, 354 (7th Cir. 1978).
Kelly contends that the Railroad Retirement Board committed three violations of her right to procedural due process. First, the administrative appeals process lasted an unreasonable length of time. Second, contrary to its own regulations, the Board obtained additional “evidence” on Kelly’s application without notifying her and offering her the opportunity to rebut the “evidence”. Third, the administrative hearing officer contacted Kelly directly to question her on her application. This, Kelly alleges, violated her right to be represented by counsel, as guaranteed by the Board’s regulations. Although each of these allegations presents serious procedural flaws, we need not reach the constitutional question with respect to the second and third contentions. With respect to those, an agency’s violation of its regulations is sufficient to taint its act. We must, however, consider the due process implications of the unreasonable delay.
III.
The Railroad Retirement Board took nearly four years from the date of Kelly’s application to issue its final decision. The Board has established a three tiered system of administrative review. The applicant must obtain an initial decision from the Bureau of Retirement Claims. 20 C.F.R. § 260.1(a) (1979). If adverse, he must then appeal this decision to a referee of the Bureau of Hearings and Appeals. Id. at § 260.4(a) (1979). In turn this decision, if adverse, must be appealed to the Railroad Retirement Board. Id. at § 260.6(a) (1979).
A situation such as this, where the administrative review process of a single disability application extended to nearly four years, is wholly inexcusable. Compare Mathews v. Eldridge, 424 U.S. 319, 330, 342, 96 S.Ct. 893, 900, 906, 47 L.Ed.2d 18 (1976) (characterizing an eleven month delay in administrative review as “torpid”, the Supreme Court held that the interests of a claimant in prompt resolution of his eligibility was sufficient to allow him to bypass the full exhaustion route.) The Supreme Court has recognized that the length of time of wrongful deprivation of benefits is an important factor to be considered in assessing the impact of government action on private interests. Fusari v. Steinberg, 419 U.S. 379, 389, 95 S.Ct. 533, 539, 42 L.Ed.2d 521 (1975) (state unemployment benefits). At least one court has held that due process was violated because the administrative determination of eligibility for benefits took too long. Steinberg v. Fusari, 364 F.Supp. 922, 937-938 (D.Conn.1973), vacated and remanded on other grounds, 419 U.S. 379, 95 S.Ct. 533, 42 L.Ed.2d 521 (1975). In that case, a delay of just over 100 days was held to frustrate due process requirements. Although there is no magic length of time after which due process requirements are violated, we are certain that three years, nine months, is well past any reasonable time limit, when no valid reason for the delay is given.
Kelly’s prolonged review process is not unique. In Parker v. Railroad Retirement Board, the agency took six years to determine that an annuity applicant was not disabled. 441 F.2d 460, 464 (7th Cir. 1971). The Seventh Circuit, finding that the decision was not supported by substantial evidence, reversed and refused to remand for further hearing to the agency because of the undue length of time it took the agency to process the disability application initially. [491]*491The Board must render its determinations with much more dispatch than has yet been exercised in order to meet minimum due process requirements.
Although the Board concedes that extraordinary and unreasonable delays violate due process, it argues that this delay is neither. The merits of that argument are patently frivolous. The Board also argues that the delay was necessitated by the need to gather additional medical evidence. The record does not support this inference; rather, after evidence was amassed, an opinion was more than one year forthcoming. The Board also seeks to justify the delay by the backlog of eases and limited resources of the Board. Whatever its internal problems, the Board has the power to implement regulations that would accelerate the agency review process. Four years is totally out of phase with the requirements of fairness.
IV.
Kelly assigns error to the Board’s acquisition of an additional medical opinion of Dr. Jakalas without informing her and giving her the chance to rebut it. This evidence, in the form of a memorandum, was independently sought by the Appeals Referee after the oral hearing and before a decision was issued. The memorandum stated:
All the medical evidence in file was reviewed. It is concluded that the appellant’s residual capacities do not restrict her from full-time light gainful activities, involving less responsibilities and tension, as already reported in my January 4,1977 memo to you and August 9, 1977 note to Mr. R. W. Gliva. Psychiatric care could help this person in her adjustment to life, but her impairment is not so severe as to preclude all work. (App. at 43).
This memorandum is substantially the equivalent of the opinion rendered by the Appeals Referee, and is also reflected in the Board’s affirmance of the opinion. The memorandum is evidence on which the decision was based.
The Board eschews all obligation to reveal evidence to Kelly and argues that the burden is on the applicant to request the full record from the Board. The Board contends that, had Kelly requested her record, the additional evidence would have been revealed.
The Board’s position flies in the face of its own regulations, which place the burden on the agency to notify an applicant, summarize the newly acquired evidence for him, and afford him an opportunity to refute it. 20 C.F.R. § 260.4(f) (1979)7
By codifying this procedure into its governing regulations, the Railroad Retirement Board recognized that “[Fjairness can rarely be obtained by secret, one-sided determination of facts decisive of rights.” “Secrecy is not congenial to truth-seeking . . .” Anti-Facist Committee v. McGrath, 341 U.S. 123, 170, 171, 71 S.Ct. 624, 647-49, 95 L.Ed. 817 (1951) (Frankfurter, J., concurring).
The Board failed to adhere to its own regulations governing its duties when it independently obtains information relevant to an application for benefits. It is a well established proposition that an agency is bound by its regulations. United States v. Nixon, 418 U.S. 683, 695-96, 94 S.Ct. [492]*4923090, 3101-02, 41 L.Ed.2d 1039 (1974); Vi-tare® v. Seaton, 359 U.S. 535, 539, 79 S.Ct. 968, 972, 3 L.Ed.2d 1012 (1959); Service v. Dulles, 354 U.S. 363, 372-73, 77 S.Ct. 1152, 1157-58, 1 L.Ed.2d 1403 (1957); Bray v. United States, 515 F.2d 1383, 1395, 207 Ct.Cl. 60 (1975). Failure to comply with regulations is a fatal flaw to the administrative action. Union of Concerned Scientists v. Atomic Energy Commission, 499 F.2d 1069, 1082 (D.C.Cir.1974). An action undertaken by an agency contrary to its regulations is illegal and of no effect. Vi-tare® v. Seaton, 359 U.S. at 545, 79 S.Ct. at 975; accord, Accardi v. Shaughnessy, 347 U.S. 260, 268, 74 S.Ct. 499, 503, 98 L.Ed. 681 (1954). Since the Board did not comply with its regulations with respect to this “evidence”, it is to be given no effect, and cannot be deemed “evidence on the record” when we review the Agency’s decision for substantial evidence.
In Matthews v. Railroad Retirement Board, 472 F.2d 1176 (5th Cir. 1973), the Board, in rejecting a claim for a widow’s annuity, relied in part on new evidence obtained after the oral hearing without giving the applicant an opportunity to rebut the new evidence. The Fifth Circuit held that this violation of the Board’s regulations required a reversal of the denial of benefits with a remand to reconsider the application after the applicant has had the opportunity to rebut. We decline to adopt the approach of the Fifth Circuit to this problem.
We see no reason why this applicant should undergo another bout in the agency. Four years is enough. Moreover, if the agency does not conform to the regulations it wrote to assure a fair hearing and an accurate determination of eligibility, then the agency should not be permitted to rely on that evidence in making its decision. To permit the agency to do it over again, this time correctly, gives the agency no incentive to comply at the start and penalizes the applicant by requiring her to return to the agency forum because of the agency’s errors. Therefore this court when reviewing the record for substantial evidence supporting the agency’s decision will disregard the “evidence” that Kelly had no opportunity to rebut. Failure to comply with its regulations renders the agency’s act null.
V.
The Board’s regulations give claimants the right to be represented by counsel in the administrative process determining their eligibility for benefits. 20 C.F.R. §§ 260.2(e)(3), 262.12 (1979).8 Kelly contends that the Board violated this right, and its own regulations, when the Appeals Referee telephoned her to continue questioning her, one day after the hearing. We agree. Her attorney was not present during Kelly’s questioning. Nor was he notified that she would be further questioned. Such ex parte questioning is inconsistent with the right to be represented by counsel and taints the decision. Although Kelly’s attorney requested that the Board refrain from contacting Kelly directly, the Board repeated its bypass of counsel and again directly contacted Kelly.
The Board addressed this issue in its opinion. It agreed with appellant that the Referee should not have contacted Kelly, and offered a “regret of this breach of profes[493]*493sional courtesy.” We conclude, however, that this was more serious than a breach of courtesy. The Board then went on to find “that this was harmless error.” We find it difficult to understand the Board’s conclusion, especially in light of the memo to the file summarizing the initial direct contact with Kelly. This direct contact was made despite the agency’s knowledge that Kelly was represented by counsel.
To begin with, Kelly was questioned about matters directly bearing on whether she was so depressed as to be eligible for a disability annuity. This evidence was taken without her gggnáel being present, although she had elected to be represented by counsel. We fail to see how the taking of evidence in the absence of counsel when the appellant has exercised the right to be represented by counsel can be mere harmless error.
Moreover, the memo to the file of that ex parte conversation includes reference to facts absolutely irrelevant to any determination of disability. The memo reports that Kelly never identifies herself when she answers the telephone because collection agencies are after her for payments. This information has no bearing on Kelly’s eligibility for a disability annuity and does not belong in an applicant’s file. Such irrelevance would not have been' extracted had the Referee respected Kelly’s elected right to have counsel present during any questioning of her. Nor would it have been preserved in her application file. We view this as a very serious breach of a fair procedure, and of Kelly’s rights, and not as the Board did — as a breach of courtesy. Therefore, because the Board violated its regulations entitling Kelly to be represented by counsel, we shall not consider the memorandum of the conversation as “evidence” in the record when reviewing the Board’s decision. That information was illegally acquired by the Board when it failed to follow its regulations and is therefore entitled to no weight.
VI.
In denying Kelly’s application for a disability annuity, the Board found that Kelly was not disabled because her condition was not severe enough to prevent her “from doing work which does not involve much stress.” See 45 U.S.C. § 231a(d)(3) (1976). In the absence of fraud, a court of appeals will not disturb the Board’s decision to deny disability benefits if it is “supported by substantial evidence in the record, when viewed in the light that the record in its entirety furnishes . . . .” Stephens v. Railroad Retirement Board, 301 F.2d 899, 900-01 (7th Cir. 1962); 45 U.S.C. § 355(f) (1976); Brotherhood of Railroad Trainmen on Monongahela Connecting Railroad Co. v. Railroad Retirement Board, 410 F.2d 353, 356-57 (3d Cir. 1969). This Court must inquire on the basis of the evidence in the record whether the Board’s decision is supported. The applicant bears the burden of adducing evidence to support a claim for a disability award. 45 U.S.C. § 23ia(d)(3) (1976). Thus, Kelly had the burden of establishing the three components to eligibility for a disabled child’s annuity: that her mental condition prevents her from engaging in any regular employment, that the condition began before the age of twenty-two, and that it will last twelve months or result in death. 45 U.S.C. § 231a(d)(l) (iii)(C); id. at § 231a(d)(3) (1976); 20 C.F.R. § 208.10 (1979). In addition, the Board may on its own initiative obtain material information. 20 C.F.R. § 260.4(f) (1979).
The medical evidence in the record consists of six professional evaluations. She was examined and treated by five psychiatrists, two of them Board certified. Both Board certified doctors reported that Kelly is not able to work. Each psychiatrist diagnosed her condition as depressive neurosis. Dr. Gnassi, who was one of the Board certified examining physicians, characterized her depressive neurosis as severe, and further stated that she had a severe cyclo-thymic personality, that is, that she has “high moods” as well as “lows.” App. at 45. Dr. Swartznburg, the other Board certified psychiatrist, reported that Kelly’s prognosis was “quite guarded.” App. at 49. He re[494]*494ported that she is “unable to tolerate the routine and ordinary demands of employment or the routine and ordinary vicissitudes of interpersonal relationships which occur in any employment situation.” “[S]he is highly unlikely to tolerate the stresses of employment in the future.” App. at 49. Dr. B. Jalali reported that Kelly’s “prognosis is fair based on the fact that she continues to receive psychiatric treatment” and stated that her symptoms at the time he was treating her were sufficiently severe to render her unable to sustain a full time job. App. at 50. Dr. Baril reported that Kelly’s emotional problems required psychotherapy. He found some evidence of depression and prescribed an antidepressant. App. at 50. Dr. M. Jalali also diagnosed Kelly’s condition as depressive neurosis and referred her for long term treatment. The only professional who examined Kelly and reported that she could work was not a medical doctor, but a psychologist, Dr. Scheiderer. App. at 45. Dr. Scheiderer was also the only person who diagnosed her condition as a “hysterical personality”, instead of “depression.”
In addition to the medical evidence on the record, the Board relied on the lay observations of the Appeals Referee as evidence in the Record. The question arises whether an administrative law judge’s untrained observations as to whether an applicant is under a disability constitutes “evidence” which the Board could rely on to support its decision.
This court has twice refused to permit an ALJ’s lay observation that a claimant appears healthy to constitute substantial evidence supporting the ALJ’s ultimate finding of physical nondisability. Lewis v. Califano, 616 F.2d 73, at 76 (3d Cir. 1980); Gober v. Matthews, 574 F.2d 772, 777 (3d Cir. 1978). Accord, Williams v. Finch, 440 F.2d 613, 616-17 (5th Cir. 1971). This rule applies with equal force to an ALJ’s lay observation regarding whether a claimant appears under a mental disability. The ALJ’s observation that Kelly appeared to be an ordinary, normal individual, runs contrary to all of the medical evidence in the record.9 Five psychiatrists classified Kelly as having depressive neurosis, one psychologist diagnosed her as having a “hysterical personality.” An administrative law judge may not reject professional medical evidence on the basis of his own observation. Lewis v. Califano, supra, at 76; Gober v. Matthews, supra, 574 F.2d at 777. Therefore, an Appeals Referee may not substitute his observation that Kelly appeared normal for the diagnoses of depression and hysterical personality submitted by professionals. As the Fourth Circuit noted in Lewis v. Weinberger, “In cases of alleged psychological disability, such lay observation [by an administrative judge] is entitled to little or no weight . . . .” 541 F.2d 417, 421 (4th Cir. 1976).
The Board found that Kelly had not made a serious effort to improve her mental condition because of her noncooperation with her therapists. App. at 67. We believe the Board’s focus to be misplaced. In Parker v. Railroad Retirement Board, 441 F.2d 460 (7th Cir. 1971), the Retirement Board concluded that Parker lacked the motivation to work and denied him a disability annuity. The Seventh Circuit revérsed, stating that it was inappropriate to focus on lack of motivation because that was one of the symptoms of the psychological disorder from which appellant suffered. Id. at 464. Accord, Branham v. Gardner, 383 F.2d 614, 633 (6th Cir. 1967). Similarly, it is inappropriate to focus on Kelly’s history of problems with her psychiatrists because one of [495]*495the symptoms from which Kelly admittedly suffers is her inability to function socially or form interpersonal relationships. Moreover, neither the statute nor the regulations require that the applicant first make an effort to improve in order to establish eligibility for a mental disability annuity.
To qualify for a child’s disability annuity Kelly need only prove that she was under a disabling mental condition, i. e., one which prevented her from working, that would result in death or last longer than one year and which began before the age twenty-two. See note 5 and accompanying text, supra. She need not establish that her condition would not improve with treatment. Therefore, the Board’s finding that with proper medical treatment her condition would probably improve, App. at 67, is irrelevant to the initial eligibility determination. It is also contrary to the medical evidence, which characterized her prognosis as “quite guarded” and “fair.”
We must now determine whether there is substantial evidence in the record supporting the Board’s determination that Kelly’s condition is not so severe that she cannot work. The evidence in the record consists of reports from six professionals who have examined Kelly. Five psychiatrists reported that Kelly was a depressive neurotic, ranging from “mildly depressed” to “severely depressed.” Additionally, one psychiatrist reported that Kelly had a severe cyclothymic personality, that is, as having recurring cycles of “highs” and “lows”. Three psychiatrists commented,, two of them Board certified, on whether Kelly would be able to work. They agreed unanimously that she could not.10 A contrary report submitted by Dr. Scheiderer, the only psychologist to examine Kelly, states that Kelly’s problems, although genuine, “do not appear to prevent her from . working.” App. 46.
Where there is conflict in the evidence in the record, an ALJ is “free to choose between properly submitted medical opinions.” Gober v. Matthews, supra, 574 F.2d at 777. Nevertheless, the decision must be supported by substantial evidence. Although substantial evidence is not easily quantified, it is “such relevant evidence as a reasonable mind might accept as adequate to support a conclusion.” Richardson v. Perales, 402 U.S. 389, 401, 91 S.Ct. 1420, 1427, 28 L.Ed.2d 842 (1971); Lewis v. Califano, supra, at 76. We cannot say that the Board’s decision which is supported by one professional report, when three other medical reports are to the contrary, is supported by substantial evidence on the record. In reaching this decision we note that we agree with the conclusion of Railroad Retirement Board’s General Counsel on this issue, who advised the Board that Dr. Scheiderer’s report “would not constitute substantial evidence . .. [to] permit a court to uphold the referee’s decision if the Board should affirm.” App. at 58. Because it does not withstand the substantial evidence on the record test, the decision of the Board will be reversed.
Moreover, the bulk of evidence in the record supports the contrary conclusion— that Kelly is disabled. This court has the authority to reverse or modify the Board’s decision without remanding the case to the agency. 45 U.S.C. § 355(f) (1976); Parker v. Railroad Retirement Board, supra, 441 F.2d at 464 (remanding solely for the allowance of disability benefits payments). Here, substantial evidence supports the determination that Kelly could not engage in gainful employment. She has satisfied the “child” aspect by establishing that her condition began before the age twenty-two. She proved this by submitting medical evaluations that antedate her twenty-second birthday. She has satisfied the “permanent” aspect by showing that her condition lasted twelve months. This she established by virtue of two doctors’ reports, whose interviews were more than a year apart, concluding she could not work. Finally, by the psychiatric reports in the record, she has established that her mental condition prevents her from working. The agency may require proof of Kelly’s continued disability. 20 C.F.R. 208.25 (1979).
We will reverse the Board’s decision without remand for further hearing, because [496]*496this conclusion does not require further evidence. Kelly has established that she satisfied the eligibility criteria for the disabled child’s annuity. Moreover, her application remained pending for nearly four years before the Agency issued its final order. Agreeing with the approach taken by the Sixth Circuit when faced with an application that remained pending an unreasonable amount of time before the Railroad Retirement Board, “[w]e see no reason to remand the case for the taking of further [evidence].” Parker v. Railroad Retirement Board, supra, 441 F.2d at 464 (application lingered for six years before the Railroad Retirement Board finally acted).
Accordingly the decision of the Railroad Retirement Board is reversed and the case remanded to the Board for the allowance of disability benefits to Kelly.