Back in 2019, during the fall quarter of my sophomore year of college, I went to my school's career fair and met a lovely gentleman named Torrey, who was representing Symantec at a modest booth. The cybersecurity firm, who at the time owned and maintained Norton Antivirus among other products, had had a stong relationship with the Department of Computer Science at UCLA on account of its nearby corporate headquarters in Culver City.
Nervous in the bustling environment, I approached the booth ready to give my best elevator pitch and drop off my resume for review. Something must have impressed them enough, because a few days later, the hiring team reached out to schedule a phone interview.
At the time, I was living in the dorms, so I commandeered my floor's lounge and got as comfortable as I could get while the butterflies in my stomach performed impressive aerial acrobatics. The phone rang and I picked up after a ring and a half. We exchanged “hello”s and “how-are-ya”s, and I realized I was talking to the same Torrey that I had met at the career fair. Nothing could have prepared me for the conversation that ensued.
Huh? I had revised tons of C++ and algorithms fundamentals, written answers to common behavioral questions, and psyched myself up to make some friendly small talk. Why was this interviewer asking me what I ought to be asked?
Still just as casually, he responded,
I had heard the advice to prepare a few meaningful questions to ask at the end of interview, but at the beginning? It simply made no sense. Sheepishly, I clarified my expectations.
He understood what was going on before I did.
Taken aback, I caught on to the fact that Torrey had mistaken me for his fellow interviewer, Hubert! The tension finally broke and we both burst out laughing. He apologized for the confusion, and I shared with them how absurd of an experience it was to come up with my own intervivew question to answer.
I hopped off the call for a bit once Hubert joined, and the rest of the phone interview continued without any further misunderstandings.
It was the best ice breaker I could have asked for. A good laugh and a memorable story goes a long way towards cementing your place once all the basic qualifications are met.
A week or two later, I was called into an on-site interview and I got to experience a corporate office for the first time, which was a lot cooler than I expected. They asked me a list question and a dynamic programming question, both of which I answered well enough to get the offer.
One thing I really appreciated about this internship was the level of organization and consideration that Symantec and specifically a few select employees devoted to it. Interns all sat together in the bull-pen rather than being cordoned off next to their respective teams, which allowed us to make friends with one another easily, and each intern was assigned a mentor on our respective team to help us onboard and get familiar with the codebase.
On my team, I was given some freedom to select tasks and projects that matched my level of interest and confidence, and I was encouraged to develop a good understanding of what I was working on rather than deliver some kind of value immediately.
The first project I tackled involved Symantec's enterprise Client-Side Intrusion Detection System (CIDS). An intrusion detection system is a piece of software that is designed to flag potentially malicious code or network traffic, after which some policy can be executed to protect data, kill processes, or simply continue monitoring a potential attack vector.
In order to keep up with the security arms race, updates to this system need to be made on a daily basis, sometimes even more frequently than that. Rather than updating the entire source code for the program, the potential attack vector signatures are encoded into a data file that ships with the program. That way, the core logic for checking traffic or code against different types of signatures stays separate from the actual list of signatures to check against, which needs those frequent updates.
The core problem that I was tasked to solve was that these updates would occasionally be very large, which cost the firm money to distribute and some small amount of compute across all the client devices running Symentec's enterprise product.
Naturally, the first thing I did was look at the data file. This proved fruitless at first, since the data was stored as raw, encrypted binary (a rather sensible decision), but after getting a local build up and running, I located the C++ code which generated the data file and temporarily disabled the encryption.
The updates were sent out as binary diff patches to reduce the total outgoing data per update, which worked on most days, but sometimes these patch files would be larger than the entire data file itself. Curious, I manually diff-ed adjacent days' generated data files using a hex diff tool.
This process was rather painstaking because I was essentially examining and parsing out the structure of a binary file byte by byte, but after reading up on some internal documentation and working with another team member, I actually noticed the pattern! One field of the signature structure would sometimes be changed across almost all signatures of a certain type.
After reviewing exactly how the data file was generated, I realized that the offending field was not stored directly, but rather as an index into a list of that data type stored near the header of the file, so whenever a signature of that particular type was added to the data file, it offset all indices which occured after that one in the sort order. To solve this problem, I rearranged the structure of the data file to include the offending field directly in the list of signatures, rather than having a list at the header into which the list of signatures indexes.
Looking back, this project essentially boiled down to “Insertion into an array is O(N), not O(1),” but that fix is still saving thousands of dollars in distribution costs every few days and reducing the size of those patches by over 90%. This was the first time I was thrown into a massive codebase, and I was very pleased with my ability to gather my bearings and implement a change that would actually make it into production on a core piece of infrastructure.
During the initial stages of the above project, I took it upon myself to learn thoroughly about how our IDS was implemented, and upon reflection, that system is pretty fundamental to many third-party, low-level softwares in the field of cybersecurity. The question I had was, “How does a program actually access the data in network traffic, in a browser, or in a syscall?” and the answer actually created my next intern project.
Turns out the adage of, “The real malware is the anti-malware software itself” kind of rings true in this case. Function hooking is a strategy for augmenting existing software by adding “hooks” which intercept the return behavior of existing library functions, thus allowing for arbitrary code execution for the purpose of detecting or preventing malicious behavior.
Yep, it's literally a white-hat rootkit exploit.
When I was poking around some of the implementation details and unit tests, I discovered some pretty strange behavior. The program would silently quit when running certain tests, and I could not for the life of me figure out why. Chasing down a silent crash in our production code? Sounds like an intern project if I've ever heard one!
This project really challenged my debugging skills, as I quickly realized how deep I'd have to wade into the world of computing. See, while the majority of the project was written in comfy C++ land (yes it was comparitively comfy), the acutal hooking was implemented with assembly. Armed with my trusty debugger, I F11 'd my way down the call stack until I could watch the stack being rearraged in real-time.
This took a considerable amount of time to understand.
Seriously, I don't know if you've ever had a project that motivated you to journal again to maintain your sanity, but it's truly a humbling experience. At one point I even considered leaving it for a more expereinced developer to figure out, only to get absolutely roasted by the architect of the hooking infrastructure. Despite the embarrasment, I knew he was right. No self-respecting engineer should leave such a low-level, critical infrastructure bug exposed.
And so I continued toiling until I found the source of the unexpected behavior: a miscalculated return address. x86 stores its return address on the stack relative to %rbp , and somewhere in the development of certain hooks, that value was retrieved incorrectly. I remember one day, as I was manually tracing the call stack, I noticed that control returned to a completely unexpected place. There was a real moment of, “Wait, I don't remember calling that function...” after which I could finally enjoy my eureka moment.
Well after my internship was over, one of my team members reached out on LinkedIn and let me know that a fix was actually implemented and pushed to prod. I really appreciated that message.
The final project I attempted at Symantec was a quality-of-life improvement for the DevOps team. We had a CICD pipeline in Jenkins that automated builds for production, ran various unit and integration tests, and deployed software to various kinds of clients. Built into the pipeline was a series of performance tests that would run automatically and raise flags on anomolous results.
My job was to take the output of this performance testing and generate a humanly readable, graphical report that could be viewed in browser. See, even though the test pipeline would automatically detect whether a proprosed change would affect perfomance drastically, there was no way to really visualize trends over time. If a piece of software was getting a small percentage worse as time progressed, we wanted to be able to catch it before it became a problem.
I'm not going to spill too much ink on this one. I can be honest enough to admit that not every intern project makes it into the codebase. Much of the actual performance testing had already been done using JMeter, so my job amounted to a bit of batch scripting, parsing in python, and general plumbing of existing tools to spit out an html. I didn't have a lot of time left in the internship to complete the job thoroughly, but I like to imagine I left a vision of how it could be done better than what I could come up with in the given time.
Eventually, the time came to start the process of offboarding and preparing our final presentations. I was pretty happy with my experience at Symantec. It was a summer of a lot of firsts for me, and I felt like my life had moved forward faster than it had ever done so before.
There's a lot of details left out of this story because I've decided to focus more on the technical details, but within those smaller, day-to-day moments, I derived a lot of meaning.
For example, my workstation was near the corner of a walkway, and I remember one day, my manager was leaving the office a bit later than usual. I happened to be at my desk trying to find a natural place to stop my workday, and I remember them questioning, “Hey Vivek, how come you're still here? It's well past 5 PM and you can always just pick it up tomorrow.” We also spent an entire two days doing a capture-the-flag (CTF) style tournament to get us interns a bit more experience with the cybersecurity field at large. There was a ping-pong table that I made use of pretty heavily after lunch many days. Some of my desk-neighbors and I even moved our stuff around to create a more open and inviting space that felt less like a cubicle. Hell, the entire company was being cleaved in twain after Broadcom acquired our enterprise team.
There's all these memories associated with this internship that don't get represented clearly in the bullet points, but they work in tandem with the technical experience I gained to create a meaningful experience. I'm glad I got the chance to get my feet wet, go deep on a couple projects, and learn more about an unfamiliar field, but at the same time, there was so much more to enjoy thoroughout the process and even outside of work. I hope I can create a lifestyle that recognizes and supports that type of multifaceted growth.