Tech Week 2020, powered by PPAI Tech Summit, is going on right now, bringing together the promotional products businesses’ IT professionals and technology leaders for an extensive range of education sessions presented by experts from inside and outside the industry. The programming also allows attendees to connect and learn from each other. Running September 21-25, Tech Week began yesterday with a presentation from an anonymous “white hat” hacker and a walk through a cyber-attack as it happens.
Security consultant Tinker Secor, a pseudonym for a hacker with extensive experience in “ethical hacking”, (testing designed to determine how an organization and its systems will fare in the event of a real attack) shared a presentation that centered around stories of actual penetration tests and breaches that highlighted the pragmatic issues addressed by security engineering and corporate security teams.
Secor began his session in the middle of a narrative, sharing how he and a partner were trying to break into a high-end retail establishment in downtown Milan. He went in as a hacker and his partner as a social engineer, to gain access to its internal network, point-of-sale devices and from there to the greater corporate network. Beforehand, he’d gotten ahold of a corporate polo as to appear to be a company IT professional, while his colleague was dressed very professionally and went in as an auditor.
Neither of them spoke much Italian, but they tried to learn as much as possible on the plane ride over. He said he memorized things like, “Hello, my name is…,” “Where is your router,” and “Please, I am unarmed, don’t shoot.” They went in with a fake audit letter supposedly from corporate IT, and said they were there to upgrade the company’s systems just before Christmas. With the letter, business cards they’d produced and Secor’s corporate polo, they were given access to the router. While he hacked into the router, his colleague identified security cameras and secured several passwords to the systems. He installed a backdoor into the system, and they left, having only been there for eight or nine minutes.
Through fascinating storytelling, Secor walked his audience through his process of penetrating a secure system. These include finding access points, usernames and even how he goes about determining passwords, and how to move through the system to go from one user’s access to the whole system. He also spoke about why people hack and their motivations, from money to ideological reasons, and what hackers do once they’re in a hacked system, from changing admin privileges to accessing web cams.
On Monday afternoon, in the session “Zero Day Attack (Cyber Incident In Progress),” Mike Pfeiffer, vice president of technology at distributor American Solutions for Business, took his audience through a cyber-attack as it happens. Pfeiffer, who has more than 30 years of relevant experience, has worked at a data and marketing solutions vendor, a consumer packaged goods company, and as a trade and expense management solutions provider. He is also the mayor of Long Beach, Minnesota.
Pfeiffer’s session related ASB’s experience with a cyber attack in April 2019, and he spoke on what the company did before and during the attack, including preventative actions that helped thwart the attack. ASB’s cybersecurity strategy is based around three tenets—protect, detect and respond—and he highlighted how the company’s actions and experiences during the attack related to them.
What is a zero-day attack? It leverages a vulnerability that is unknown to, or unaddressed by, those who should be interested in mitigating the vulnerability—including the vendor of the target software—and is being actively exploited in the real world. Following is Pfeiffer’s account of the details.
At 5:55 pm, the company received a down alert from its WebLogic web server. In an effort to detect attacks, ASB uses monitors on all servers to detect a down condition immediately. It uses another application, AppDynamics, to monitor response times on its servers and databases, and the tool is so efficient, they are frequently able to alert integrated vendors of issues with their web servers before they are aware.
At 5:57 pm, ASB received a DNS alert that its public IP address was no longer available. The company knew at that point it was under attack, as the earlier issue could have been a server issue. Protecting the infrastructure with the right people, processes and tools can mitigate issues, as well as establishing a production change control process (PCCB) for clearer internal communication.
At 6 pm, as part of its response to the attack, ASB contacted its on-call rotation of network and application staff and activated its cyber-attack “Go Team.” A Go Team should include a technical lead—cyber or technical lead; a communications lead—an IT executive, a role filled in this instance by Pfeiffer; an application owner—an IT/business lead; an app developer—to serve as a technical application resource; and a IT manager/system admin—that runs the restore infrastructure.
By 6:05 pm, as part of its response, ASB was assessing the modified directories on the application server, and its admin noted they had encrypted files on the web server. Pfeiffer pointed out, “That really changed the flavor of the event.”
By 6:24 pm, staff had detected the source of the issue and commenced cleanup and containment. They identified the zero-day exploit as a WebLogic server vulnerability.
At 6:42 pm, they received a firewall alert on malware. It was a retroactive detection alert. The ransomware being placed on their system was also a zero-day payload, meaning it was not recognized at the time by detection software. The alert at 6:42 pm signified the various detection tools recognizing the payload as malware. Pfeiffer said when you get a zero-day exploit and zero-day payload, this was as bad as it gets.
At 7 pm, research began to locate the files needed for the restore. ASB’s use of the principle of least privilege (PLP) in setting up accounts stopped the attack’s progress through their system. PLP refers to the concept and practice of restricting access rights for users, accounts and computing processes to only those resources absolutely required to perform routine, authorized activities. This prevented the attack’s lateral movement through ASB’s system, which would have created a much bigger event. At this part of the team’s response, it killed network access to the targeted environment and used out-of-band access to assess the environment.
At 7:14 pm, they started download of the restore. The National Institute of Standards and Technology’s incident response process contains four steps: preparation, detection and analysis, containment, eradication and recovery, and post-incident activity. ASB’s recovery actions included blocking the entire range of IP from the origin area, closing the zero-day WebLogic exposure and updating definitions, and restoring the environment.
At 7:30 pm, ASB’s IT team communicated the incident to senior leadership and the affected user community and anticipated recovery. In communicating its response, the team emphasized its performance on the recovery time objective (RTO) and recovery point objective (RPO) metrics. RTO is how long it takes to recover a production environment and RPO is how much data are you subject to lose.
At 8:04 pm, zipped backup files were downloaded. At 8:26 pm, the unzip finished and copy to webserver started. At 8:53 pm, copy of files was finished. What are some of the lessons learned from this process? Pfeiffer said that they improved their backup appliance to cut restore time in half for future events and they added further privileged reduction and network segmentation roadmap items.
At 9:02 pm, the web server was back online and the system was accessible. The RTO was three hours and no data was lost. Pfeiffer outlined what really worked in their response: the least privileged account use, the clear and calm communication between team members and their predefined roles, as well as their known and tested RTO and RPO.
At 9:13 pm, the team issued an all-clear that the system was operational. At 9:23 pm, they restored files to the second web server. At 10:17 pm, they communicated to the leadership team that redundancy was achieved. At 10:30 pm, they started rebuilding the third web server. At 11:09 pm, they announced that they were at full strength with all servers available.
Tech Week continues through Friday. For the schedule and to register for the event, click here. Everything—except for Secor’s session—PowerPoints and supporting documents will be accessible on the platform for registered attendees at least through November.