In the 1999-2000 academic year a score of colleges and universities declined to participate in the survey by Yahoo Internet Life, a Ziff-Davis publication, that underlies the magazine's annual Wired Campus feature. Several other institutions asked YIL to be excluded from the feature even though they had already submitted data, something YIL refused to do. These notes sketch the context for last year's decisions by the nonparticipating institutions, plus events and discussions since then.

The reasons institutions chose not to participate last year varied, but certain themes were consistently discernible in discussions online and at conferences last winter.

1. Some institutions simply do not participate in surveys like this, regardless of the topic, except in certain exceptional circumstances.
2. Some institutions believe that surveys and rating exercises focused on a single narrow institutional attribute disserve both individual institutions and higher education generally. They therefore participate only in survey/rating exercises that cover a substantial array of institutional attributes.
3. The survey questions appeared, to some institutions, to embody a particular definition of successful undergraduate education generally, and of student-oriented IT in particular. They appeared to reflect a belief on YIL's part that research IT is irrelevant to undergraduate education, for example, that residential students are more important than commuting students, and that online contact with student systems and course materials is better than personal contact with administrators and faculty.
4. The survey itself was sloppily executed, with ambiguously worded questions, puzzling answer categories, and other incentives for aggressive rather than fair responses from institutions. YIL and Peterson's provided little guidance to institutions that sought guidance on ambiguous questions, and issued no general notices to correct misleading questions.
5. To explain certain extraordinary changes in its ratings, YIL accused several institutions of lying in their responses to earlier surveys, yet it implemented no auditing or verification procedures to minimize such a problem.
6. YIL refused to disclose the process through which it translated survey data into ratings, and indeed the full dataset itself. Since this rendered YIL's ratings irreproducible from data, it gave rise to the strong suspicion among some institutions that the ratings reflected the subjective judgments of YIL's editors, at least in part, rather than the consistent application of formulas to data.
7. The YIL ratings are immensely volatile from year to year, far more than actual changes in relative IT investment or adjustments to formulas can possibly explain. Much of the year-to-year variation thus appears to be random, which may reflect error or close scoring or editorial adjustment or all of them, yet YIL makes no effort to explain or deal with this.
8. YIL did not adjust its ranking procedures to account for growth in the survey universe, and did not provide sub-ratings for different kinds of institutions. The net effect of these practices was to make the expected change in ranking from year to year for the average institution downward, an unpalatable prospect for most.

At the suggestion of several campus CIOs, YIL explored the possibility of a session on its survey at this fall's Educause conference. Although Educause offered prime time for such a session and arranged for one of its board members to moderate it, YIL refused to participate unless it could have substantial control over the discussion and assurance that it would not become a "gripe session". Separately, in reponse to separate suggestions that YIL empanel a group of knowledgeable individuals to assist with survey and rating design, YIL argued that it should  seek advice on the survey only from individuals unconnected with colleges or universities, since anyone connected with a college or university would be interested in advancing his or her institution's standing and therefore could not be "impartial".

Over the past few days representatives of several colleges and universities, both participants and nonparticipants, have discussed these issues with YIL editors and representatives of Peterson's, which does the actual data collection. In these meetings YIL evinced interest in improving the survey instrument itself, so as to minimize ambiguities and biases in the questions. YIL also plans to be somewhat more open about how it constructs ratings, but it maintains (as of 10/11/00) that it will not disclose its methodology. YIL continues to argue that rating colleges and universities unidimensionally by their IT activity is a good thing, that the magazine's pedagogical and philosophical biases belong in its ratings, and that disclosing its data or methodology would undercut its effectiveness.

On the basis of last winter's interactions and discussions since then, it appears that most of the institutions that did not participate in the YIL survey last year will again not participate this year. In addition, several institutions that participated last year now plan not to participate, including, according to conversations at Educause, many CLAC and COFHE colleges and universities.

notes compiled by GA Jackson <gjackson@uchicago.edu>,
who also maintains the list of current non-participants.
13 October 2000