Background
By day, I am a full-time Information Security professional with nearly 20 years of industry experience. I have worked for companies as small as a few dozen employees to some of the largest organizations in my state and other mid-to-large sized companies in between. About 5 years ago, I answered the call to give back and began to work in the evenings as an Adjunct Professor in a local Cyber Security program. It has been rewarding to work to train the next generation of the cybersecurity workforce.
Early in my tenure, my campus fielded a Cyber Security Competition Team. Throughout the year, there are competitions where youngling cyber practitioners are given real-world challenges and opportunities to get some priceless experience. The other coaches of this team and I are proud that most of our students get job offers on the spot from companies of all sizes and industries based on their performance in these competitions. To date, our job placement rate is near 100%.
The following contains lessons learned that I had the opportunity to observe and experience firsthand throughout the genesis and maturation of a Collegiate Cyber Defense Team. As a highly experienced and credentialed security practitioner, I am still amazed at how many companies struggle with cybersecurity. I share the following, hoping to show many that good security doesn’t have to be hard or expensive and that even old security dogs can learn new tricks.
Lesson #1. No plan survives contact with the enemy.
“Everyone has a plan until they get punched in the mouth.”
-Mike Tyson
Any theory can be good until it hits the battlefield. When we first started, we followed the methodology that when you have a compromised box, the first thing you should do as a part of containment and eradication is identify the missing patches and quickly apply those patches to keep the Threat Actors (TA’s) out. In theory, this works and makes sense. Patch Management is a part of good hygiene that every network should implement. However, we only have six hours to complete our mission during the competition. We don’t have endless days post-detection. In this environment and conditions, we must change our focus.
Initially, our teams were quickly getting onto their perspective boxes and attempting to download the patches, wasting hours they didn’t have. The result was a constant seesawing back and forth with the Offensive Team (Red Team)* because they could not properly contain the systems using this method. So, in this situation, rushing to patch as a mitigation step just didn’t work. So that led us to change how we were doing things going forward.
Our methodology shifted to applying patches at the very end of the process versus being one of the first things our teams do. Now, our teams focus on getting control of their boxes and removing the threat actor. Typically, they just find the sorts of Indicators of Compromise (IOCs). Once the student has maintained network control over their system, other mitigation steps and compensating controls are deployed. This further decreases the pressure for patching. From year to year, we saw a rapid increase in the performance of the team by then focusing on the core containment and eradication rather than on the remediation steps. Patching is still important. It is just not as important as we had initially thought. Compensating controls can equally mitigate the vulnerability, often much quicker than patch deployment. This leads us to our next lesson.
Lesson #2. Do the basics and do them well.
“Ransomware Threat Actors are very good at the basics, and this shows against companies that aren’t.”
-SANS DFIR Aug 2023
Everyone has heard of the KISS (keep it simple, stupid) process method. This is especially true when defending your system against a strong, active, offensive adversary. Many students who join and compete on the Cyber Team have little to no real-world IT experience. They spend their weekends preparing for the competition by reviewing basic checklists and tasks. These exercises are done repeatedly, just like practicing free throws in basketball.
At one of our most recent events, a student on the team was new to managing a Linux BIND server. When the starting gun fired, almost immediately, the entire network services went down. This is not uncommon in these sorts of competitions. Students in all positions on all teams struggled to get going and get their services back up and running. This student, who was new to BIND, followed their training and worked off the checklists and task lists. Shortly after that, the BIND server was back up and running and remained up for the entire competition despite a constant barrage from the Red Team. This student’s work was recognized during the competition. This student received public credit from judges and a merit award for their work. When the student was asked by other students what they did to get working while others still failed, the answer was that they just stuck to the checklists. While others tried kitchen sinks and chased sexy tactics, this student “KISS’d” the Red Ream goodbye. Doing the basics well can be its own no/low-cost mitigation tool. This leads us to our lesson.
Lesson #3. Network control is a force multiplier.
“Don’t be afraid to challenge the pros, even in their own backyard.”
-Colin Powell
You rarely get some security theories exercised in real life. Observing these competitions has allowed me to see some theories put to the test. To give some context, the network environment is prepared before the students can log in and begin. The Red Team pillaged the network the night before, leaving their rootkits and shells behind. Often, systems are further damaged to hamper the response efforts of the students. I think this is a bit much. The Red Team already has the advantage of network reconnaissance and unfettered access to plant their tools. It is unfair for some of the country’s best Red Teamers to get to further damage systems, handicapping the rookie students who are outclassed in skills and experience. I’m not alone in this sentiment. The other coaches on our team agreed. We decided to take it to the Red Team to prove that none of their tools would work if we controlled the network.
In our first attempt, we fully leveraged all aspects of the Layer 7 Firewall from Palo Alto that was being utilized in this round of the competition. (In each round, different technologies are used, such as PFSense or Cisco). We had one very enthusiastic student buy into what we were trying to do. He took Lesson #2 to heart and learned the basics of that security appliance. On competition day, within the first hour, this student completely evicted the Red Team from the network and kept them out for the duration of the competition. He showed that a properly configured and fully utilized Layer 7 firewall is tough to bypass, even by the pros who more than had a head start. Our success led to future competition rule changes, including the increased use of legacy traditional firewalls with fewer capabilities. We proved that someone with the right security tool can properly defend themselves against a literal army of invaders with Cobalt Strike. Score one for the home team.
Now that we had made a name for ourselves, future rounds were met with the wrath of the Red Team. Brittle egos are slow to mend. Whenever the Red Team wanted to handicap the field, they ensured that any firewall, other than a Layer 7 firewall, was used. This proved that our concept was sound, but now we had to adapt. The battle line had moved.
We moved the control of the network to the host level. In our basic checklist and task lists, we included steps to manage the endpoint firewall manually, whether it was IPTables or Windows Firewall. Students drilled on locating the connections to their system and evaluating whether it was truly needed. In other words, we implemented Zero-Trust networking a little before it became a marketing craze. It didn’t matter what was protecting the perimeter; the hosts were actively rejecting connections from sources that were not allowed. Eventually, the Red Team would pivot to a trusted host to launch an attack, but the time was running out by that point of the competition. Time and again, when our students achieved control of their host and managed the local firewall properly, the Red Team faced greater difficulty in achieving their goals, even on servers with multiple vulnerabilities. In a nutshell, any server regardless of Operating System, even if out of date and running with critical vulnerabilities can still be thoroughly secured without negatively impacting business. This leads us to our next topic.
Lesson #4. Know the rules—all steps of the process matter.
“If you can’t describe what you are doing as a process, you don’t know what you’re doing.”
-W. Edwards Deming
This lesson comes from one of the first national competitions where the team placed very strongly in second place. The distance between first and second place was measured in milliseconds. Many of the other teams in the final round had what I referred to as ringers. While our students were all in associate and bachelor’s degree programs, other teams fielded students who were not only in graduate-level degree programs but who often worked full-time in IT/IS already. These teams have stronger technical acuity and can sprint harder during these timed competitions.
Leading up to the competition, our coaches reviewed all the documentation on how the events would be scored. After all, this is still a game, despite the attempts to emulate real-world security events. All sections were covered for review, including a new section. The judges would quiz each team on the Change Management Process in this new scoring section. While change management is an important aspect of any information security program, it doesn’t have the flashy appeal that battling an active adversary does—blocking and tackling matters. We took the time to teach the Change Management Process to the team thoroughly.
During the competition, our team took some lumps in the first round. We were back in the pack. Later in the day, while one team was in the hot seats taking on the Red Team, the other teams were quizzed on Change Management. Our team responded well to the quiz and took 1st place in that aspect of the competition. They leapfrogged their way to the final four. During the live finale, the team came up just milliseconds short of taking first place. Without the preparation of all security processes and not just focusing on active defense, the team delivered an overall strong security performance. In the end, this is the overall goal to begin with. This is something that many companies need to emulate—all steps in the information security process matter. Oh, and the team got their due when they returned in 2022 and won the competition, besting 1100 other national teams. This leads us to our final topic.
Lesson #5. Microsoft Defender is actually really good at what it does.
“Microsoft is not about greed. It’s about innovation and fairness.”
–Bill Gates
I remember when Microsoft Defender Antivirus, formally known as Microsoft Defender for Endpoint, formally known as Microsoft Defender Advanced Threat Protection, formally known as Microsoft Defender, formally known as Windows Defender Antivirus, formally known as Windows Defender, formally known as Microsoft AntiSpyWare, first debuted. I was at Microsoft Tech ED 2008 in Orlando. I sat in on a couple of presentations of what the developers were trying to do with the product. It wasn’t just a Symantec knockoff. It was original from the ground up.
Fast forward 15 years later. As previously mentioned, during the first stages of the competition, the Red Team gets access to the network to plant their tools and handicap the students. During one round, all the Windows servers (2012 and newer) had Defender disabled and sabotaged to prevent it from running during the competition. During the daily debrief, the captain of the Red Team talks about their observations. When asked about Defender, he admitted that the Red Team intentionally gutted Defender. “If we had let it run, most of our tools wouldn’t work.” That strongly indicates how good this “out of the box” security product is. It drives Red Teams to sabotage it to achieve their objectives. The Offensive Security Team at SecurIT360 highly regards this product in their testing.
Don’t take their word for it. Check out this blog referred to as “Last Antivirus Standing”. Java Exploit Code Obfuscation and Antivirus Bypass/Evasion (CVE-2012-4681) (security-obscurity.blogspot.com). The author took an 8-year-old JavaScript virus and in minutes, made the most subtle tweaks to get it past 43 of 44 antivirus engines with relative ease. The lone antivirus that detected the virus during every mutation was Microsoft Defender. Long story short, Microsoft Defender is a good, solid security solution, especially for those with smaller budgets. Anyone telling you that Microsoft Defender is garbage is either financially incentivized to tell you something different or doesn’t know what they are discussing—most of the best Red Teamers out their respect Defender as a good defensive tool. The facts are there. I was fortunate to see it put through testing that most people don’t. Leveling the playing field against offensive adversaries seems like a pretty fair move to me. Thanks, Bill, and company.
Wrapping up
Teaching and coaching cybersecurity has been enlightening and rewarding. Although the victories were celebrated, watching the students grow in skill and confidence was the true honor. As mentioned, I was fortunate enough to have the opportunity to witness firsthand the battles waged by a Collegiate Cyber Defense Team against some of the country’s best Red Teams. Lessons gleaned from those clashes were shared here in hopes of helping others who are on the frontlines of today’s cyber battlefields defend themselves. Our program succeeded on a wafer-thin budget compared to competing schools with 10x to 30x the resources. The lessons shared here show how any company, even those facing the same fiscal challenges, can improve security, one small step at a time.
And yes, the tears of Red Teamers are delicious, even if they are very short-lived.
SecurIT360 is an independent, vendor-agnostic technology company focused on developing programs and systems specifically catered to our client’s needs. While some vendors are listed here, we work with each customer and their selected IT solutions on a custom basis and not “one-size fits all” approach.
* In the context of cybersecurity, a “Red Team” is a group that pretends to be an enemy. A “Blue Team” is a term for cybersecurity team who are responsible for defending networks and computers against attack. Red team – Wikipedia. https://en.wikipedia.org/wiki/Red_team.
