As told by Nancy Beck, Systems Management and Operations

The latest episode of "Design on a Dime: Data Center Edition" aired on January 5th and 6th, when a crack team of UAlbany staff, along with a few token IBM reps, tore down and rebuilt the hardware that supports the AIX Peoplesoft and MyUAlbany infrastructure in less than 30 hours.  The seamless execution of this undertaking is a testament to the team work, coordination and meticulous planning that preceded the actual event.
 
This episode is just the most recent installment of a multi-dimensional and ongoing data center upgrade project. For the past several years, it has been apparent that the University's IT needs are outgrowing the capacity of our data center facility. While it's true that a plan is in motion to provide a formal assessment of our current facility and make recommendations both for its rehabilitation and for alternative solutions, new datacenters do not spring up over night and millions of dollars don't fall from trees.  Therefore, interim steps must be taken to squeeze every drop of useful life out of the "little data center that could," in order to keep its shortcomings from inhibiting the growth of IT on campus, and by extension the growth of the University.
 
To this end, research began into data center best practices.  Armed with useful information and advice, a team of hopefuls began to devise a plan to implement as many solutions as possible in our existing environment, in effect "picking the low hanging fruit."
 
In an earlier episode - an out-of-code, disorganized electrical system was addressed by the replacement of all panels and wiring throughout the main data center.  After putting that fire out focus turned to the next item on a list of pressing issues: cooling, or lack thereof.
 
With two of the three cooling units in the main data center in excess of 25 years old, it's not a surprise that the lack of adequate, reliable cooling was of critical concern.  With the assistance of our partners in Facilities Management, we were able to add a new air conditioner and are in the process of upgrading all plumbing for both the primary supply which is fed from our glycol towers, and the backup which is supplied by domestic water.  These improvements to the cooling infrastructure are only part of the solution.  There were other contributing factors.
 
The "Hot and Cold Row" story...
 
The pace of IT growth on campus has pushed the data center to its limits.  For the last several years, each new rack was expeditiously installed in whatever free space was available to save time and prevent service interruptions. This ultimately resulted in a poor use of space and contributed to our inefficient use of physics to assist in cooling the data center. To further compound the issue, the data center uses a raised floor system designed to provide a plenum for the transfer of cool air from the down-flow cooling units to the server racks.  Over the years the sub-floor provided a convenient place to conceal the extensive cabling system.  The congestion under the floor grew to a proportion that impeded airflow.  Moreover, the sub-floor environment can be less than ideal for delicate communication cables and its inaccessibility makes troubleshooting difficult.

A solution presented itself in the form of an overhead cable tray.  A re-deployment, overhead, would not only improve our troubleshooting capability, it would enable us to improve documentation and make upgrades easier. While it's true that some don't consider it the most visually appealing option, the tray provides a safe, accessible path for cables to run overhead freeing the space beneath the raised floor for its intended purpose: airflow. 
 
An idea was born and the plan to reorganize the data center began to materialize using a model referred to as "hot and cold rows".  The principle is simple.  All racks are oriented so that the fronts face each other leaving the backs to follow suit.  Since air is taken in the front of every server for purposes of cooling, it makes sense that delivering cold air, via vent placement, directly to the front of every rack in the "cold rows" would enable us to cool more efficiently.  Likewise, heat from the rear or "hot rows" rises to the ceiling and is transferred back to the cooling units. 
 
Using Visio we began to lay out our proposed floor plan.  There were many versions as impediments and obstacles were overcome one by one, until finally the layout was complete (on paper that is).  If only the racks could be moved as easily in real life as they had in Visio world...

Prior to the rack moves, a mini-project was undertaken.  During the days of the mainframe Operations staff and printers occupied the same room as the equipment, as their job required physical interface and close proximity.  While that is no longer true in our environment, the Operators and printers remained in the machine room.  The environment required, and created by, the servers and peripherals is less than ideal by human standards. Furthermore, datacenter best practices make recommendations that apply here.  One - minimize the frequency and duration of human interaction in the server room.  Two - provide separate locations for servers and printers.  And three - segregate printers from human occupants.

The path was clear.  We needed to relocate the staff, printers, and the storage cabinets, removing them from the server room, and giving each their own space.  Multiple benefits would be gained.  It would open up space on the server room floor, which was essential to allow for the first rack move.  It would also provide a pathway around the perimeter for movement and set up of large equipment.  Finally, it would provide a separate space for people, in a more appropriate environment.  An environment that includes more comfortable temperatures, less noise, limited exposure to electro magnetic fields and of great value...a place, a desk, a drawer, to call their own.

A process began which left us with a nicely segregated server/machine room, office space and printer/storage space.  This mini-project took a lot of time and effort but only laid the ground work which allowed us to move on to the real challenge.

The idea of a reorganization of the machine room, complete with installation of an overhead cable tray, was pitched.  In an effort to make the concept more manageable, the decision was made to break it down into two phases, the first of which would focus on the west half of the room only.  The idea took root and grew into a project.  Then the real work began.  Extensive, detailed planning and estimating commenced for the multiple areas to be addressed including: cable tray, electrical work, connectivity and vendor support.  Subsequently, the meetings began, but these meetings were a productive collaboration between many groups - System Administrators, DataComm, UAD, Operations, and Facilities Management - all working together to accomplish a shared goal.
 
Along the way, we decided to add a much needed maintenance of the core router and an implementation of "in-row switching" to the mix, knowing we needed to take advantage of any downtime, a precious commodity in the 24/7 IT environment.  The core router upgrade would provide greater network reliability and the capability to statically switch to a backup supervisor, without human intervention, in the event of a failure.
 
Chris, our fearless leader, went to battle and returned victorious, securing a weekend's worth of downtime during which our projects would unfold.  And that they did. 
 
Day One, January 5th - Beginning at 3:00 pm, a systematic process requiring the expertise of each group began.  Backups were completed, followed by shutdown, power-down, cable disconnect, equipment moves and last but not least, final electrical work.  All wrapped up at approximately 2:00 am.

Day Two, January 6th - Team members, along with IBM engineers, were back bright-eyed and bushy-tailed, at 6:30 am to begin the rack moves, which were complete by 10:15.  At 8:00 am, part of the DataComm team began work on the core router while others executed a well-choreographed cable reconnection.  It was like... ballet on ladders.  Soon Systems began trying out the new neighborhood.  By 5:30 pm UAD joined the dance with start-up and testing of the Databases and Applications.  By 8:30 pm all reports indicated we were well on our way to being a fully functional entity.

 
The project start to finish was nearly seamless, and complete well short of our deadline.  Planning, and lots of it, accounted for much of our success.  However, the biggest lesson we should all take away from the experience is that it takes a village.  When we work together, across organizational lines, we can move mountains (or at least a whole lot of computing equipment).
 
This was such a positive experience it almost took on the feel of a celebration.  We needed the knowledge and the experience of each and every team member and each and every team member gave freely of their talents. 

I personally could not be more proud.

GO TEAM!