The Technical Services TeamTop of Page

Primary Responsibilities – Technical Services Unit – currently organized into five teams.

1. Transmission Infrastructure
2.Telecommunications
3. End User Computing
3. Systems Infrastructure
4. Network & Security Infrastructure
 

Technical Services Organizational ChartTop of Page

Technical Services Unit Major ProjectsTop of Page

The Technical Services Unit has several large and multi-year projects that started in 2016 and continuing into and through planning cycles to achieve a 5-year refresh plan.  Projects include: Expanding district network capacity to the Internet, Expanding network capacity to the colleges, Creating reliable network links between the colleges and district datacenters, redesigning district-wide network and connectivity architecture, upgrading district-wide wireless networking infrastructure, replacing core and distribution switching infrastructure in LRCCD datacenters, replacing core and distribution switching infrastructures at all LRCCD colleges, modernize the district data centers with SmartRow technology, and document and enforce district-wide infrastructure standards.  The Technical Services Unit successfully completed 10Gbps CENIC connectivity to the DO and FLC Datacenters, Skype for Business VoIP rollout, 10Gbps Firewall replacement at the DO and FLC Datacenters including Intrusion Prevention System replacement, Fax Server system implementation, Centralized Wireless Controller Upgrade using an N+1 approach to service continuity, and the removal of the 20+ year old NEC PBX telephony environment.

Technical Services Unit Assessment Results, Gaps, and InitiativesTop of Page

2015-17 Accomplishments and Status

2015-2017 Technical Services Unit Assessment Results, Gaps, and Initiatives

In the Spring of 2016, the Technical Services Unit was assessed externally by the CampusWorks assessment team.  This represents the second year that the Unit has engaged in both internally and externally conducted assessments of overall Information Architecture, Network Services, Customer Service, Security, IT Alignment and Organizational Structure, and the overall Technology Environment as it relates to the Student Experience.  Subsequent dialogue produced general consensus on the existence of significant operational risks and meaningful gaps between where we believe we are and where we believe we need to be.  The gaps and risks were reviewed by DOIT leadership and reviewed with College IT staff.  It became clear that several gap/risks stood out from the others as requiring special attention.  CampusWorks placed the gaps in the category of “Infrastructure, Network, and Security.”  CampusWorks focused their recommendations on Resources, Resource Allocations, Project Management, and Governance (Roles & Responsibilities differentiation between College IT staff and DOIT staff).  Many of the CampusWorks recommendations will need to be addressed at levels above the Technical Services Unit. For the Technical Services Unit, we are using six areas of gap/risk as focal points as they were consistently identified in all assessments.  The first three gap/risk areas, Backup & Recoverability Architecture, Wireless Architecture, and Datacenter Architecture were investigated and scoped using internal resources and were conducted by DOIT staff in conjunction with College IT staff and Facilities Management staff.  These assessments were later validated by the CampusWorks Second Opinion Technology Assessment.  Below is a high-level summary of critical gaps and their resulting initiatives.

  • System Backup & Recover-ability Architecture – significant gaps (All Resolved)
  • District – Our confidence in our ability to recover systems from tape is very low (resolved)
  • District – Archival tape systems at end of useful life (resolved)
  • District – Backup & Recovery systems and storage capacity inadequate – some production systems excluded from Archival process (resolved)
  • College – Backup challenges exist at multiple colleges (resolved)
  • College – Disparate Archival systems across colleges (resolved)
  • Wireless Architecture – significant gaps – (All Resolved)
  • District – Wireless AP’s at or nearing end of life (resolved)
  • District – Controller software issues and aging AP’s negatively affect service (resolved)
  • District – Wireless vendor change required due to product line removal from higher education market – (new standard selected (resolved)
  • College – Wireless implementations incomplete at every college (resolved)
  • College – Wireless service not consistently meeting the needs of students or faculty (resolved)
  • Datacenter Architecture – Major Critical Gaps – (in process)
  • District – No single Datacenter w/capacity to maintain District Operational load
  • District – Primary datacenter condition exposes district to excessive risks of failure (Water, Power, HVAC, cabling, etc.) (in process)
  • District – Secondary datacenter similar to primary however co-located with FLC equipment and regularly intermingled with FLC equipment and services (in process)
  • District – No datacenter with adequate environmental controls or adequate power infrastructure (in process)
  • District/College – Construction/Remodel Standards (e.g. Physical Layer, Environmental controls, Power, UPS, Generator, etc.) not consistently utilized

Several urgent projects or initiatives remain unresolved from the earlier assessments.  Examples of these projects and initiatives include but are not limited to the following:

  • Replace Core Networking Equipment (in process 50% Feb 2018)
  • Replace Motorola wireless with new Cisco wireless at each of the colleges (resolved )
  • Expand college wireless service and coverage areas resolved )
  • Partner with Facilities Maintenance department to update existing datacenters with SmartRow or similar technologies (in process)

The next three risk areas were assessed externally by technology professionals specializing in key areas of Network Architecture and reliable network service delivery.  The Technical Services Unit contracted with Shandam Consulting for a professional technical assessment of the LRCCD Network Core Architecture, Internet Connectivity Architecture, and Firewall Architecture.  Below is a high-level summary of findings, recommendations, and resulting initiatives from each of the assessments.

  • Network Core Architecture Assessment
  • Summary of Findings
  • Obsolete core network devices (solution resolved implementation in process)
  • Network devices listed as “End of Life” by manufacturer
  • Lack of visibility into network performance metrics (resolved)
  • Three disparate monitoring platforms – unclear monitoring strategy
  • Many single points of failure throughout architecture (WAN resolved)
  • Where redundant power supplies exist – they are frequently connected to the same circuit
  • Manual configuration management and unmonitored configuration changes using shared device credentials
  • Quality of Service strategy not meeting end user needs (in process)
  • Existing switching platform has limited QoS options adversely affecting network bandwidth management options
  • High Availability strategy is incomplete – Layer 2 focus (solution resolved implementation in process)
  • Switch uplinks are oversubscribed by a ratio of 48:1 (more than twice the best practice ratio)
  • Summary of Recommendations
  • Replace core networking platform and equipment (ARC, FLC resolved, SCC and CRC in process)
  • Redesign fiber WAN connectivity – provide redundant connections to datacenters (DONE)
  • Improve network core fault tolerance with redundant trunked links and server links (DONE)
  • Deploy redundant core switches with interconnected multiple trunked 10 gigabit connections to provide layer 2 & 3 fault tolerance (using a link aggregation design) (DONE)
  • Upgrade switch uplink connections to 10Gb (Mostly DONE need new fiber needed to meet standard)
  • Consider a new core switch layer utilizing virtual switching technologies (DONE)
  • Implement Layer 2 management features (STRG, BPDU and Port Security) (DONE)
  • Implement Layer 3 management protocols (VRRP, VRF) and Layer 3 diversity (DONE)
  • Upgrade QoS and increase circuit bandwidth (DONE)
  • Enable Netflow and use a Netflow collector (DONE)
  • Upgrade/Replace existing power distribution units and UPS’s and add redundant power supplies to key network equipment – using a redundant circuit model (DONE)
  • Create better, more complete documentation to be used in configuration management and troubleshooting (in progress)
  • Develop a formal process of identifying, tracking and deploying software across network devices (DONE)
  • Develop a network monitoring strategy with a single, vendor agnostic monitoring system (DONE)
  • Deploy a Configuration Management tool to enable auditing, archival and automated alerts (DONE)
  • Summary of High Level Projects and Initiatives relative to this assessment (DONE)
  • Work with CENIC and the CCCCO to increase our CENIC connectivity
  • Work with CENIC and the CCCCO to build alternate connectivity to each college
  • Review alternate fiber connectivity between colleges and the LRCCD datacenters (DONE)
  • Implement 10 gigabit Firewalls at DO and FLC datacenters (DONE)
  • Implement Intrusion Prevention System at DO and FLC datacenters (DONE)
  • Replace datacenter core network infrastructure (in progress)
  • Replace datacenter network distribution infrastructure (DONE)
  • Partner with American River College to Architect, Engineer, Acquire, Build, and Deploy a Student Centered, Service Focused network (DONE)
  • Explore extended DOIT involvement in college switching and UPS replacement cycles (DONE)
  • Develop Centralized Budget for Switching and UPS replacements (DONE but Tech Plan changed direction)
  • Partner with American River College to replace their wireless network (DONE)
  • Explore Network Access Control solutions (in progress)
  • Explore Network Security Assessment
  • Leverage new networking capabilities with a redesign of our network Architecture (DONE)
  • Engineer a Highly Available, Robust and Manageable network with QoS and traffic shaping on both private networks and our public network (in progress)
  • Define Configuration Management Strategy and operational objectives (in progress)
  • Explore Configuration Management Tools and define criteria for product assessment (in progress)
  • Define Network Monitoring Strategy and operational objectives with escalation (in progress)
  • Explore Network Monitoring Tools and define criteria for product assessment (in progress)
  • Internet Architecture Assessment
  • Summary of Findings
  • Top 10 Technology Concerns from LRCCD College IT Staff:
  • Bandwidth Limitations to the desktop, between buildings, to datacenter limit success of technology initiatives
  • Network Transparency and visibility into network performance, utilization, security, and data flows is nearly universally desired
  • Many Single Points of Failure and overall lack of Redundancy / Fault Tolerance for key circuits and networking equipment
  • Inadequate Wireless Network does not meet college wireless needs
  • Voice over IP (VoIP) call quality and fault tolerance on installed network infrastructure
  • IT Support Staffing Levels create single points of failure and are inconsistent with the diversity of the environment
  • Intra-College Communications, and input from college IT staff with user community involvement need to inform technology policy decisions
  • Clarity of District Information Technology Strategy/Vision and the consistency of its communication
  • Standards for New Technology Implementations and onboarding
  • Network Access Control policies, procedures, mechanisms and tools necessary to protect LRCCD networked data and assets
  • Overall - Lack of available bandwidth for core applications
  • Bandwidth to the Internet is congested during business hours causing contention between critical business applications and non-critical services
  • Overall - Lack of visibility into network performance metrics
  • Three disparate monitoring platforms – unclear monitoring strategy
  • Overall - Many single points of failure throughout architecture
  • Not all Colleges have redundant physical pathways to the Internet
  • Single Internet service provider (CENIC)
  • The Active/Passive circuit connectivity model leaves 50% of the available bandwidth inaccessible 100% of the time
  • Overall - Single person dependencies in key IT roles
  • Geographically and Technically diverse IT environment with key dependencies in a few critical roles
  • Overall - High Availability strategy is incomplete
  • Active/Passive Internet connectivity through a single provider
  • Overall Assessment of current IT Management Maturity - Reactive
  • Summary of Recommendations
  • Implement fully redundant (and active) Internet connectivity
  • Load Balance Internet traffic across redundant links
  • Replace VRRP failover strategy with Boarder Gateway Protocol (BGP)
  • Re-engineer internal traffic patterns and network prefix advertisements
  • Expand Internet bandwidth capacity at the border and between colleges
  • Utilize network monitoring tools that track real time statistics as well as averages over time through re-defining and re-engineering our monitoring granularity
  • Identify, document, communicate and measure Acceptable Outage Intervals
  • Apply and monitor quality of service (QoS) policies and their effect on traffic
  • Consider/Evaluate the addition of a secondary Internet Service Provider connection
  • Obtain Service Level Agreements (SLAs) with all service and transport providers where ever possible and re-evaluate relationships/contracts where SLAs are not possible.
  • Expand and formalize on-the-job cross training and an internal “mentoring” process
  • Firewall Architecture Assessment
  • Summary of Findings
  • Firewall failover strategy does not meet business needs (Resolved)
  • Single person dependency in firewall support role (Resolved)
  • Lack of comprehensive reporting and logging capability (Resolved)
  • Lack of visibility into firewall performance metrics (Resolved)
  • Many single points of failure throughout architecture (Resolved)
  • Summary of Recommendations (not previously identified above)
  • Replace the existing firewall platform (Resolved at Datacenters, ARC, and FLC)
  • Consider Information Security Officer (ISO) level review of broad firewall policies and change management methodology
  • Re-architect firewall solution to consolidate to a single platform and to as few devices as necessary
  • Implement and utilize a log correlation engine with High Risk event notification
  • Ensure reverse logic monitoring points exist and return desired negative responses
  • Implement a Configuration Management System that meets industry standard notification and auditing requirements
  • Ensure Remote Authentication Protocols allow for access tracking and reporting

For more than 10 years, only one of the districts four colleges has had an independent connection to the Internet. Each of the other colleges is dependent upon leased fiber connections back to the district office data center for their access to district resources and the Internet. In some cases, a single strand of fiber is the only thing connecting a Los Rios College (e.g. CRC) to our Network. Assessments conducted over the last two years have identified critical gaps in our ability to deliver services to students.

As identified last year, the Technical Services Unit, in collaboration with the colleges, must design, architect, engineer and implement a secure, reliable, performant and effective network that meets LRCCD business objectives and is integrated to the highest degree possible to ensure both the effectiveness and efficiency of maintenance and support activities.  These necessary network upgrades will not be limited to services and equipment alone as business practices, maintenance cycles, operational tasks, support activities, and overall manageability, performance, and reliability are dependent upon the solutions chosen. Affected areas of the network include: Internet connectivity, routing, firewalls, switching infrastructure at all levels, wireless services, and all voice, video, and data services.  The Technical Services unit approached and successfully petitioned the State Chancellors Office and CENIC for improved connectivity.  Please see the attached “Connectivity Update” for details on the CENIC primary connectivity approach as well as information on the RFP for Secondary connectivity options.

The team will continue to work with third-party networking consultants to ensure that LRCCD business objectives are the focus and foundation for technical solutions. The results of these two efforts will form the foundation for LRCCD to build the technological infrastructure necessary to meet the needs of our students today and to be fit to compete in the future.

Data operability and recoverability are key topics both for the district office and for each of the LRCCD colleges. The Technical Services unit recently expanded the storage area network in our primary and secondary data centers to address system performance issues. They also acquired, installed, and configured both the backup and recovery disk solutions and the backup and recovery software solutions necessary to meet LRCCD business objectives. The next step was to secure, equip, and connect an appropriate “third-site” to meet district-wide data protection and disaster recovery needs. The district’s disk-to-disk-to-tape model has been successfully replaced with a disk-to-disk-to-disk model with multiple options for both continuity and recoverability including the use of remote third site storage.

DOIT Tech Services Project CalendarTop of Page

  • Tech Services Project Calendar
Calendar
Calendars