By Erin Hurley
On Jan. 25, 2024, the campus network experienced a 6.5-hour intermittent outage, rendering it inaccessible; this is a frustrating yet recurring issue. Students, staff and faculty were unable to utilize the internet on campus, inhibiting many functions. Campus network outages are not uncommon, and people often do not appreciate the stability of the network until it is gone.
“Our campus network should be just as stable as the lights in the building,” says Francisco Chavez, the University’s chief information officer. You may recognize his name from email communications received regarding network outages and planned maintenance. He was appointed in Feb. 2023 as the new CIO for the University after a rigorous recruiting process, and has been investigating recurring difficulties faced by Technology Services since he took office. Chavez transitioned here from St. Mary’s College of California, another liberal arts institution in the San Francisco Bay Area. He explains that the challenges our campus is currently facing are problems he addressed at St. Mary’s 15 years ago, and though it will take time, Chavez is confident the challenges can be fixed.
One week after Chavez was appointed last year, the campus faced an unfortunate network outage. After this failure, he requested the Infrastructure and Operations team to perform a full review of the existing hardware to diagnose and fix the problem. In this process, it was discovered that there is infrastructure as outdated as 30-years-old. “We actually did a survey, you know, where we went to every single building, thousands of fiber strands, you know, and some of them would just deteriorate in your hand,” Chavez says. From this investigation, a three-phase plan was adopted to completely renovate the existing infrastructure and ensure less frustrating outages in the future.
Yet, as proved by the Jan. 25 outage, the difficulties have not been completely solved. Since last summer, Technology Services has been working to upgrade the infrastructure, but they are stalled between Phase two and three of the plan.
Outages and network irregularities seriously impact students, who rely on a stable connection for their work. Sophie Haynes (‘27) expresses ongoing frustration with the network. They explain how difficult it is to get stuff done and that “obviously it’s hard to do homework or classwork, but then it’s also hard to access things like the printers or other services that are based off of the campus wifi.” The outages and unreliability have widespread effects.
An additional hindrance to the official outages is the suboptimal functionality of the network on a day-to-day basis. Students may have to connect to their own personal hotspots or attempt to use the guest network in order to access their devices. Failing that, the next option is to find somewhere off-campus that offers wifi. Haynes has a car on campus and is able to move her workspace to a coffee shop or other place with a reliable connection during outages. “But I definitely know that it’s a privilege to be able to have my own transportation and be able to do that,” they acknowledge.
Chavez explains that various upgrades to internet infrastructure have occurred during different administrations over the decades. He compares the cluttered equipment to that of a cake assembled by different bakers. “You have a design that doesn’t really align together. It’s just different expressions of how they felt the network should be,” he explains. Tech Services has to work to untangle many years of incompatible, challenging infrastructure, which takes time. “The outage that we experienced is related to an old network topology that we have yet to deconstruct and repair.”
Chavez explains that it is ideal for institutions to do a complete overhaul of the network every seven to ten years. Unfortunately this has not been occurring, which is why network outages are common. “We can approach it in different ways where we build in the replacements over time so we’re not dealing with a mountain of replacements, but what seems to be more palatable for the University is just you know, saving money for the capital replacement over the seven-year, ten-year time span and doing a full lift like we’re doing now.” Previously, smaller upgrades were prioritized due to funding concerns. But as the department was focused on addressing immediate problems rather than tackling the root cause, they were unable to save money to finance larger overhauls of the infrastructure. One part of Chavez’s three-phase plan is to replace the switches that connect devices together over the network, allowing them to exchange data. To put these costs in perspective, each switch is priced around $2,000 – $3,000, and replacing a whole network of them can be upwards of $2.5-3 million.
Fortunately, the department had enough cash reserves to finance Phase one and two of the “network infrastructure refresh project intended to address the deferred maintenance [of] our campus network,” Chavez explains. This includes the replacement of firewalls, core data center switches, and distribution switches, as well as the implementation of a Virtual Extensible Local Area Network. These projects were completed, but Phase three, which includes refreshing the hardware that connects users to the network, is currently unfunded. Chavez says that he has spoken to President Crawford, and network stability is seen as a top priority on campus. “Hopefully, we can keep that momentum and start working on deploying that third phase of our upgrades.”
The campus might be looking at multiple months of rocky connection as Technology Services works to replace infrastructure that may be upwards of a decade outdated. These outages and the overall temperamental state of the network make many aspects of life on campus difficult. Haynes framed the importance of wifi perfectly: it: “just permeates every single aspect of campus life, whether we want it to or not, so having working WiFi (sic) is really important for having a working campus.”