You can't go back, only forward.

Post date: Sep 2, 2016 1:27:04 AM

tl;dr: I will no longer be developing squert. If you want a little history and the reason, read on.

A little over ten years ago I sent this message to the snort-users mailing list:

I initially built squert because I needed to get alert data into the hands of non-analysts so that they could identify infected clients and fix them. At the time I was responsible for the security of a semi large and strangely complex network which I was going to do my best to try and protect. Unfortunately, few people shared this interest and I only had about 5$ in my pocket. 

The existing Intrusion Detection user interfaces available at the time were either unusable or designed and built for people that had a background in security; specifically, people with incident handling and response abilities and that knew how to deal with the alerts when they came rolling in. I had a handful of generalists that weren't particularly interested in having something else dumped on their plate.

I needed something that was unambiguous in its message (think traffic light) and that was easily accessible via software that everyone would very likely already have installed on their computer. With these requirements, no real plan and no formal knowledge of web applications I built a very rough but working prototype in a couple of weeks. Squert version 0.1.0 was born.

Beautiful isn't it? 

It didn't matter, we started finding bad stuff immediately and very slowly managed to clean up a very messy network. The biggest win though was that once you have a way to easily articulate a problem; once you have data about the problem and can display it simply, it's hard ignore it any longer. 

We finally had visibility into our environment and with this visibility came a requirement to react.The earlier versions were very simple but I was always looking for ways to make the traffic light better.

GEO

Geo was probably one of the first things I started playing with. The actual implementation was super ghetto, it consisted of a small TCL script that would query the RIRs, format the results and toss them into a DB so that I could perform joins on data. I always wanted to improve this feature but never got a chance to return to it.

First attempt at GEO presentation

Toying around with a tag cloud type idea (click would filter by)

This was further refined with added functionality in later versions and ultimately fed back into several of the charts and views that you will see below (hover would show details, click would filter)

CHART JUNK

Concurrently with GEO I was always trying to display the data in as many different ways as possible. I tried a LOT of stuff. I played with so many different libs, chart types, different data, different aspects of the data, and composites all to see what worked and what didn't

I always loved this effect. It can be quickly processed. Not bad for a bunch of <tr><td>s

I wasted too much time playing with link charts. It was a complete dead end for me. I could never come up with consistent enough results that gave me confidence in the story that the data was trying to tell

The moment I saw a Sankey Diagram though, I had to have it!

In later versions I was trying to provide as much context as possible in a single view. I didn't want people to waste cycles on something that if by providing a secondary piece of information was enough for them to make a quick decision and move on

The default aggregate alert view was a perfect example of this strategy. Each signature line contained numerous disparate pieces of data (some composites) with the hope of telling a story before actually looking into the alerts detail

And when you finally did drill down into the alert you could get even more detail and a history

This isn't a chart but I loved this feature. Once an object was flagged it was trivial to pick out in later views

The last thing I was working on; integrating data from Bro so that you could pull in this extra source of data right under the event you were looking at

SO WHY STOP?

When I initially built squert it was intended to be a tool for generalists. I didn't know what I was doing or how to write software, I was simply trying to solve an immediate problem I had. When the project ultimately landed in the Security Onion linux distribution all of this changed. All of the sudden people actually started using it, something I had never anticipated.

Almost overnight, there was a user base (some of them analysts) and with this new user base came a quiet expectation that the tool needed to be more aligned with the analyst console Sguil. The complexity of this tool coincidentally happened to be the reason that I needed to create squert in the first place.   

The attention to the project and the feedback from people using it pushed me want to make it the best tool that I could. I knew that there were people that were trying to defend networks that had no resources; people that relied on tools such as this because they really had no other options. People that were in the same position that I was when I started to Google: How do I make a web application? I couldn't pack it in just yet.

Unfortunately, maintaining an open source project is work. Even when I was committing somewhat consistently on the project I was doing so in tiny spurts whenever they were available. There was never a sit down, make a plan; OK this is where are we going next! It was more like: I have an hour, what can I add that will make this view pop, how can I make this better. No plan. Just drive. 

You can't go back, only forward. 

If I didn't subscribe to this there would have never been any feature adds, no toying with new ideas, no trying out some new wizbang lib. I would have simply spent all of those short spurts cleaning up bad code which I never really learned to write, when all I wanted to do was move forward. Well, that's just no fun :)

if you have been following along closely you may have picked out that I am passionate about this stuff. I like to help wherever I can and am willing to sacrifice my own time and effort to do so. 

I have learned so much in the past year and now I want to go back. I want to fix all of the things that I know are wrong or could be done better but it is just too much. It would require a huge rewrite and I don't even use the tool anymore.

@bammv Thanks for putting up with all of my questions

@securityonion Thanks for complicating things ;)

@mephux Thanks for being a solid mentor