The client's Industrial Control System (ICS) network was critically lacking in redundancy, with multiple pieces of equipment reaching end-of-life (EOL). Additionally, the network faced a significant vulnerability due to a single point of failure in its connection to the upstream routers, which posed a serious risk to the reliability and security of the entire network.

Our team undertook a comprehensive approach to address these issues. We identified all existing end-of-life systems and mapped out the entire network using packet captures (PCAPs) and switch configurations. This allowed us to create an accurate asset inventory and connection points across the network. Through this process, we discovered several undocumented devices and connections, which were then integrated into the updated network inventory.

Once the current network state was fully understood, we developed a detailed network diagram highlighting all switches and their connected endpoints. This diagram served as a critical tool for visualizing the existing vulnerabilities, particularly the single points of failure. Collaborating closely with the client’s IT and operations teams, we drafted a new network topology that removed these vulnerabilities. The new design incorporated redundant connections between critical switches and rerouted traffic paths to ensure no single device or connection would compromise the network’s overall integrity.

In parallel, we thoroughly analyzed potential replacement hardware, considering factors such as compatibility with the existing infrastructure, long-term support, and enhanced security features. After selecting the appropriate hardware, we facilitated the procurement and installation of new, upgraded switches to replace the outdated EOL systems. Our team coordinated with the client to ensure that the implementation of the new hardware was phased, minimizing the impact on ongoing operations. Each phase was carefully planned and executed to avoid any interruptions, ensuring that the client’s critical systems remained operational throughout the upgrade process.

The project encountered several significant challenges that required strategic problem-solving and flexibility. The first major challenge was the discovery that the existing asset inventory was either missing or incomplete. Many devices were undocumented, and the configuration details of several key network components were outdated or inaccurate. To overcome this, our team had to manually verify and document every asset in the network. This was a time-consuming process, but it was essential to ensure that the network diagram was accurate and comprehensive.

Additionally, the limited documentation of the existing network layout posed difficulties in planning the upgrades. Without a clear understanding of the current network, there was a risk of unforeseen complications during the upgrade process. To mitigate this risk, our team leveraged PCAPs and switch configurations to map out the network's traffic flow and device connections. This data-driven approach allowed us to build a reliable picture of the network’s structure, but it also highlighted the complexity of the task ahead.

The criticality of the systems involved added another layer of difficulty. The ICS network supported essential operations, meaning that any downtime or misconfiguration could lead to significant operational disruptions. Careful planning and precise execution were therefore essential. We developed a phased implementation plan that allowed for the gradual introduction of new hardware and redundant connections. This approach minimized the risk of outages, but it required meticulous coordination between our team and the client’s operations staff. Timing was crucial; each step had to be executed during carefully scheduled maintenance windows to ensure that the running systems were not impacted.

Despite these challenges, the project was successfully completed with minimal disruption to the client’s operations. The new network design eliminated the single points of failure, significantly enhancing the redundancy and reliability of the ICS network. By transitioning to a collapsed core architecture, we created a full mesh of connections between all access layer switches and the distribution layer switches. This architectural shift not only increased the network’s redundancy but also optimized traffic flow, ensuring that no single switch failure could disrupt the network’s operations.

The replacement of EOL equipment with modern, robust systems further ensured the long-term sustainability and security of the network. With this new, resilient design, the client now benefits from a well-documented and robust ICS network capable of supporting their critical operations with reduced risk of failure. The full mesh topology also provides greater flexibility for future network expansions or modifications, making it easier to maintain and adapt to evolving needs.