For more than 12 years, I’ve been organizing and running hackathons with the goal of finding security vulnerabilities and fixing them before a product hits the market. These events can play a pivotal role in the product development lifecycle, increasing team collaboration, uncovering problems, and driving innovation. In this article, I’d like to share some of my key insights and tips that could help your organization create or refine a security hackathon playbook.
Why run a security hackathon?
Hackathon events bring together product and security experts for the sole purpose of finding security vulnerabilities within a product. The security experts provide the security mindset and knowledge about how to break and hack systems, while the product experts provide knowledge about the inner workings of the specific target product.
While these events are traditionally short term (lasting only a few weeks), doing them successfully requires defining a clear scope of the evaluation. This might be an architecture or design review, pre-silicon or post-silicon evaluation, looking at an entire platform or device, or a specific aspect or feature of a product.
When running a hackathon, there are traditionally four primary goals or objectives.
First, is for it to serve as a supplemental security evaluation. Second, is to build a community of practice that transfers knowledge among the participants. Third, is an organizational assessment of security capabilities and quality of Security Development Lifecycle (SDL) execution by product teams. And the fourth is to reduce costs by developing and cultivating experienced internal security research teams that can reduce dependency costly external consultants.
How do you organize a hackathon?
Here are some tips to consider when creating your playbook:
1. Don’t overlook critical details in the planning process – Prior to a hackathon, planning is key. This includes locking in a commitment from both the product and security teams participating, assigning roles, and understanding where in the product or project lifecycle a hackathon makes sense.
It’s also important to confirm the availability of key individuals for different areas such as product architecture, design, and security research. The planning goal is to have a mature enough product for security evaluation and testing, but also have enough time to make any significant changes if deemed necessary.
Equally important is locating space, equipment, tools, and infrastructure to support the evaluation activities. And finally, be sure that all participants are familiar with the hackathon methodology and process before the event starts.
2. Define the hackathon roles – These events have several defined roles that must be filled by the planning team prior to kick off:
- Two leads (I recommend one from the functional team and one from the security team) that are responsible for organizing the event. These individuals can also play additional roles if needed.
- A security lead responsible for analyzing the security architecture and identifying the priority security areas.
- An enabler responsible for setting up the testing environment.
- A facilitator that is responsible for guiding team activities during the event.
- Security experts, which, depending on the product being evaluated could be one or many people across hardware, firmware, software, network protocols, etc.
- Functional experts which are product team members versed in the functional components of the product.
- Available on-call resources such as architects, developers, validators, infrastructure experts, and others.
3. Start the security team from a blank slate – Understanding the methodology and process is different from knowing the actual details of the product being evaluated.
Product knowledge is not required for security researchers, but for complex products and/or technologies, I highly recommend the security team ramp up their understanding of product architecture and design as much as possible. This may mean spending a bit more time during the architecture and design training phase (which is one phase of the larger event that also includes a security evaluation and testing phase, and a final day wrap-up phase).
An understanding of what has been tested by the SDL team is a plus. After the security team or sub-teams are familiarized with the architecture of the product, they can begin the security evaluation by brainstorming a list of attack scenarios and test cases and applying them.
4. Prioritize targets of evaluation – These events have limited time, which means you must prioritize your targets based on the highest business risk. This could be areas that have not been heavily evaluated by the SDL team, areas that could have a significant security impact, or end-to-end platform solutions that were not evaluated in IP or component level SDL.
5. Mandate daily sync-ups – To make sure that the entire hackathon team is making positive progress, a daily sync-up should be mandated. During that sync-up time, the sub-teams report out their status and explain what tests they have performed on the product, what vulnerabilities have been discovered, and what test plans they have for continued testing on their assigned product area.
Sub-teams should be encouraged not to create an exploit for any vulnerability that is discovered, and to keep exploring other attack vectors if they get rat-holed into a test case that is not progressing well.
6. Make time to document evaluation work – These events can fly by and it’s important to capture the progress being made by teams. I recommend participants frequently note the small things, details, to consolidate them at the end of the event.
Be sure to reserve half of the final day for participants to document and report out on the security evaluation work that’s been done. These reports can be written by individual participants or by sub-teams. The intent is to ensure that the security analysis done during the event (for example tests executed, tools used, etc.) is captured and incorporated into a final hackathon report that can be used for any future security evaluations.
7. Follow up on items and close the loop – Not all items get completed during the hackathon. For example, an activity might require more test cases or inquiries as part of the brainstorming. Be sure these outstanding items are assigned to owners and activities completed. This will be essential for the final report.
8. Rate the vulnerabilities – Confirmed security vulnerabilities need to be reviewed and rated for severity using CVSS scoring.
Both the product team members and security researchers should aim to come to an agreed upon severity rating of all issues found. Only issues that have a confirmed security impact and exploitability path should be rated as security vulnerabilities (others can be noted as recommendations for security enhancements or Defense in Depth, or open sightings that need further investigation post event).
9. Produce a final report of findings – Following the hackathon, team members should deliver findings to the product lead for a final report. This includes details around all attack scenarios conducted, discovered vulnerabilities and any recommendations for improvements. Two primary expected outputs include identified issues and security recommendations, and the long-term impacts for improving security (such as threat model gaps or SDL quality of execution).
10. Evaluate the hackathon process for improvements – Running effective events can be a complex process and there is always room for improvement. Conduct a retrospective of not only the process but the findings. This can be used to drive systemic improvements in product security architecture, design, and the evaluation process. That may include adoption of security tools by product teams and the addition of security countermeasures that address classes of vulnerabilities identified during the hackathon.
Hackathons can be fun, engaging, and drive innovation when organized and executed properly. As you work to launch or improve your hackathon events, consider some of the key insights I’ve included above.