Higher Ed in 4k: Nov 2020 Report
Introduction
In November 2019 and then again in November 2020, Pope Tech conducted an accessibility evaluation of every top-level .edu domain in the United States listed in the IPEDS database.
The accessibility evaluation was performed using the Pope Tech platform which is powered by the WAVE testing engine.
Background
The Higher Ed in 4k Project was inspired by the WebAIM Million Project. After the WebAIM Million project launched, one of our takeaways was the opportunity of projects like this to help make websites more accessible in a significant way.
With this in mind, we asked ourselves:
What would happen if we went deeper than the home page, if we focused on one group (Higher Education in the US) and let the institutions rescan their data on demand?
What if we gave them free access to automated accessibility testing to document their progress on a live website?
The Higher Ed in 4k Project is more of a living experiment than a one-time study.
Methods
For context, we have included the key summary points of the methodology:
- Approximately 4,000 (4k) higher education institutions from IPEDS list were included
- Up to 100 pages (all within 4 links from the main home page) were sampled from each institution
- The accessibility evaluation was performed by Pope Tech using the WAVE engine
- Automatically detectable WAVE errors (Errors & Contrast Errors) were used to calculate the average errors per page for each institution
- The institutions average errors per page were used for the analyses and rankings
- We rescanned the same pages in 2020 that were scanned in 2019
- Web pages or websites that were missing or invalid were dropped from the 2020 analyses
A more comprehensive description of the 4k Project methodology is provided in our Nov 2019 Report.
2020 Results
Overall, Higher Education has historically done better than most industries with regards to web accessibility. While many industries and business sectors have been slower to respond to improving web accessibility, higher education continues to improve. Our 4k Project documents the improvements made by higher education institutions in the United States between 2019 and 2020.
Key 4k Project findings (Overall)
- In 2020, 5,642,199 detectable errors remain (an average of 20 errors per page)
- 8.16% of pages have no detectable errors
- The 2020 results had an average of 4 fewer errors per page than 2019 results (a 17% decline!)
- Alerts stayed nearly the same between 2019 to 2020 with average of 27 alerts per page each year
- In 2020, 7 institutions had 0 detectable errors on the pages we sampled (+2 from 2019)
- Comparing 2019 to 2020 results, the sampled pages for 63.8% of the institutions decreased in the average errors per page
Our 4k Study results are consistent with WebAIM’s reported trends for higher education. WebAIM found that while non-higher education websites tended to get slightly more errors per page between 2019 and 2020, higher education pages improved across the same time span.
Key Take Away Points
- Higher education tends to do better than other industries in web accessibility
- There was a 17% reduction in the average errors per page between 2019 and 2020.
- Just under ⅔ of the higher education institutions made progress on detectable errors on the pages we analyzed. This means, however, that over ⅓ of the institutions did not make any progress or regressed.
The "Average" Institution (2020)
The average institution in the 2020 data set had the following:
- 75 pages scanned
- 20 errors per page
- 27 alerts per page
- a user would encounter an error on 1 in every 40 page elements
5 most common errors
| Error | Count | % of Errors |
|---|---|---|
| Very Low Contrast | 3,737,103 | 66.1% |
| Empty Link | 673,602 | 11.9% |
| Missing Alternative Text | 457,340 | 8.4% |
| Missing Form Label | 221,687 | 3.9% |
| Empty Button | 147,393 | 2.6% |
It is important to realize that if the 5 most common errors detected in this project were resolved, it would remove 93% (5.3 million) of the detectable errors from the pages we scanned.
Concurrently with this report, we released a blog about the most common errors detected in this study, as well as well as suggestions on how to fix them.
State/territory rankings
For the second year in a row, Alaska has the lowest amount of detectable accessibility errors with only 7.8 errors per page, followed by Hawaii, Wyoming, Alabama, and Montana which were all below 13.5 errors per page. While these states tend to have lower population levels, the overall state population didn't correlate with the number of errors. Rhode Island and Pennsylvania were right at the average with 20 errors per page.
The 5 states with the highest average number of accessibility errors across their higher education institutions were Colorado, Kentucky, New Mexico, South Dakota, and Utah with over 25 errors per page each. Forty-seven of the 50 states average error counts across their higher education institutions decreased.
When we initially calculated the state-average scores, we did so using the mean (average) of all the institutions in that state. . We discovered that there were some outlier institutions with very high error per page counts that were skewing the results. Based on this, we used each state’s median institution’s errors per page as the average for the state rankings.
On our State Rankings page you can find each state’s ranking, median, and errors per page. We provided two ranks, an overall ranking with all institutions (public and private) and a seperate ranking for just public institutions.
The State and institution results are updated in real-time as institutions rescan and (hopefully) improve their web accessibility.
Rankings by classifications
Using the IPED List data, and other higher education institution databases, we also were able to compare institutions based on different classifications beyond just overall rankings.
Below are some of the interesting comparisons we found.
Highest Degree Offered
| Type | 2019 Errors per page | 2020 Errors per page | % change |
|---|---|---|---|
| up to Associates degree | 24.8 | 20.0 | 19.6% decline |
| up to Bachelor's degree | 25.1 | 20.3 | 19.1% decline |
| Post Bachelor degrees | 23.3 | 19.4 | 16.4% decline |
Private vs. public institutions
| Type | 2019 Errors per page | 2020 Errors per page | % change |
|---|---|---|---|
| Public | 20.21 | 15.4 | 23.8% decline |
| Private not-for-profit | 25.3 | 21.2 | 16.0% decline |
| Private for-profit | 30.1 | 26.2 | 12.9% decline |
Public institutions averaged 15 errors per page, and have fewer accessible errors per page than private institutions. Similarly, Private not-for-profit institutions have fewer detectable errors (with 21 errors per page) than private for-profit institutions which have 26 errors per page.
It would make sense that public institutions would be better as they have additional laws beyond the Americans with Disability Act including Section 508.
Student enrollment
| Type | 2019 Errors per page | 2020 Errors per page | % change |
|---|---|---|---|
| Enrollment under 1,000 | 28.3 | 24.1 | 14.8% decline |
| Enrollment 1,000 - 4,999 | 24.1 | 19.2 | 20.4% decline |
| Enrollment 5,000 - 9,999 | 19.0 | 15.4 | 19.3% decline |
| Enrollment 10,000 - 19,999 | 19.0 | 14.3 | 24.7% decline |
| Enrollment 20,000 and above | 14.0 | 11.2 | 20.2% decline |
There was a direct correlation between student enrollment numbers and accessibility errors per page. The more students, the fewer detectable errors.
This could be because the larger institutions have more resources and budget. Institutions with over 20,000 students enrolled had only 11 detectable accessibility errors per page.
Land Grant
| Type | 2019 Errors per page | 2020 Errors per page | % change |
|---|---|---|---|
| Land Grant institutions | 15.4 | 13.8 | 10.4% decline |
| Non-Land Grant institutions | 24.3 | 19.9 | 18.1% decline |
Land Grant Universities had fewer detectable errors (13.8 per page) than non-land grant institutions (19.9 errors per page).
In a system
Many higher education institutions are part of a state or private school system with many intuitions in the school system.
| Type | 2019 Errors per page | 2020 Errors per page | % change |
|---|---|---|---|
| In a system | 23.1 | 17.8 | 22.9% decline |
| Not in a system | 24.6 | 20.8 | 15.4% decline |
Institutions in a school system had fewer errors per page (17.8) and resolved a higher percentage of issues between 2019-2020 (22.9% decline) than institutions not in a school system (20.8 errors per page and a 15.4% decrease in errors).
Carnegie classifications
When comparing Carnegie classifications, it is important to understand that not all institutions had a classification in the IPED List. The results only reflects those with a specified classification. Additionally, we only included a subset of the Carnegie classifications.
| Type | 2019 Errors per page | 2020 Errors per page | % change |
|---|---|---|---|
| Doctoral/Research Universities--Extensive | 15.3 | 12.4 | 19.0% decline |
| Masters Colleges and Universities I | 19.1 | 15.4 | 19.4% decline |
| Baccalaureate Colleges--Liberal Arts | 20.8 | 15.4 | 26.0% decline |
| Associates Colleges | 21.5 | 17.2 | 20.0% decline |
| Baccalaureate Colleges--General | 24.4 | 19.3 | 20.9% decline |
| Medical schools and medical centers | 27.8 | 19.8 | 28.8% decline |
| Schools of law | 30.7 | 32.1 | 4.6% increase |
Doctoral/Research Universities had the fewest accessibility errors averaging only 12 errors per page. It is interesting that the two Carnegie classifications with the highest number of detectable errors were Medical schools with 20 errors per page and Schools of Law with over 32 errors per page. Of the different Carnegie classifications, only Schools of law increased in average errors per page from 2019 to 2020 (a 4.6% increase). All the remainder classifications we evaluated had declines in detectable errors.
UCEDD vs. non UCEDD
UCEDD stands for University Centers for Excellence in Developmental Disabilities Education. The vision of the UCEDD program is, "a nation in which all Americans, including Americans with disabilities, participate fully in their communities." There is at least one in every US state and territory housed inside a host university.
| Type | 2019 Errors per page | 2020 Errors per page | % change |
|---|---|---|---|
| UCEDD | 13.8 | 11.9 | 13.8% decline |
| Non-UCEDD | 24.3 | 19.9 | 18.1% decline |
The host UCEDD institutions have just under 12 detectable errors per page, which is 8 detectable errors per page fewer than the average higher education institution (19.9 errors per page).
Institutions who requested their 4k account
Between November 2019 and November 2020, eighty-seven higher education institutions took us up on our offer to access their 4k project account in our platform. This has allowed those higher education institutions the opportunity to review their institution’s results, fix issues, and even rescan to update results. The updated results are included on our live project page and rankings are recalculated.
Did schools with 4k project access improve results?
One question we had was would an institution having access to their 4k project account accelerate the decline in errors compared to other institutions?
One interesting finding was that institutions that requested their 4k accounts tended to already have fewer errors than their counterpart institutions (14.6 errors/page compared to 24.1 errors/page in 2019). This likely suggests that institutions that requested these accounts were already more focused on accessibility than their peer institutions.
This focus on accessibility was also reflected in their overall decline in errors (a 24% decrease) between 2019 and 2020. In contrast, institutions that didn’t request access to their accounts only had an average 17% decline over the same period of time.
It is difficult to know if participation in the project was responsible for the larger decline or if we simply documented progress that would have already occurred at these institutions.
With that being said, we do think that having access to the project likely made some difference. We were able to anecdotally observe a number of these institutions that requested 4k access were using their accounts and regularly scanning within the first few weeks of access.We also noticed for some of the higher volume 4k account users that they had some impressive declines in detectable errors.
Whether or not the 4k account only helped facilitate a process already occurring or helped stimulate new initiatives is beyond the scope of the data we currently have.
We hope, of course, it was the latter. But even if it was just documenting their progress, given the frequency of scanning they were doing in their free, 4k project account, we believe that they found value in using the platform.
Interesting and random tidbits
In 2019, we reported on some other interesting findings from our project. Below, we have included our 2020 updates on each of these different findings.
Skip Links
In 2020, we found there were 1,991 skip links that didn't have a target across the sample of pages in our project. This means that someone went through the effort to add a skip link but then either never tested it or it was broken with a template update. The good news here is this was a substantial decline from the 6,964 skip links without targets we detected in 2019 on the same pages.
Marquee tags
In 2020, in our sample of pages, there were 46 marquee tags still around (a substantial drop from the 170 we detected in 2019).
Layout tables
We found 296,891 layout tables (in 2020) compared to 335,183 found in 2019 and 47,810 data tables (in 2020) compared to 53,715 found in 2019. A data table is classified as a data table if it is a properly structured table with the proper heading rows. Realistically, we suspect that there are relatively few true layout tables.
PDFs
We found 619,755 links to PDFs in 2020 or just over 2 per page (similar to 2019). These may or may not be accessible. But as we know from the 2019 WebAIM screen reader survey, 74% of screen reader users are either Very Likely or Somewhat Likely to encounter significant issues accessing a PDF document.
Conclusion
While there is still significant work to be done to ensure Higher Education websites are accessible to everyone, we are encouraged by how much better Higher Education websites were this year compared to last year. Collectively, they did resolve over 17% of the average errors per page. Additionally, nearly ⅔ of the institutions had improvements in their overall error/page averages. This suggests that the majority of higher education institutions are making efforts to improve web accessibility.
We are hopeful that this project and other endeavors by the Higher Education web accessibility community will continue to bring more awareness to web accessibility. are optimistic that over time we will see additional improvement.