5 ms·
Good timing, I just left a company after being on their security incident response oncall rotation for 2 years, partly due to the oncall. akg_67 has some great
by bobmagoo 12y ago
Good timing, I just left a company after being on their security incident response oncall rotation for 2 years, partly due to the oncall. akg_67 has some great points (https://news.ycombinator.com/item?id=9011293 https://news.ycombinator.com/item?id=9011293), but I'll add a few of my own:
1) When you're oncall, your time and priorities are no longer your own.
At your kid's soccer game? A date night? Planning on doing any of those things? Be prepared to get pulled out at any moment to deal with something that could take hours to resolve. This was the part that really got to me. As much as I'd like to do any one of those example things, I had made a prior commitment to be available and had to honor that.
2) Know the response time and physical location requirements for responding to a page
Is this something you can just fire up your laptop and an aircard and jam on, or do you have to be able to drive to the office within an hour. Don't forget about driving through places with less than great cell phone coverage.
3) It can be fun
There was a part of me that really liked the adrenaline rush of getting paged in on a legitimate security issue and having to run the call and pull the right people in to get the situation handled. It's a great test of how well you know the environment and where all the pertinent information lives.
4) Know the team size and oncall frequency
akg_67's estimate was spot on. Anything shorter than a month is crazy and you never quite feel like you normalize. Since it's based on team size, know what the optimal size of the team is and that there's funding for it? My team imploded and at the end there were only a few of us on the oncall rotation. Bear in mind that oncall duty doesn't go away because you no longer have the staff to make it manageable.
5) Vacations and sick time are now more complicated
Who has to be oncall during Christmas/4th of July/etc? What used to be some loose coordination with your manager is now a give/take discussion with your team about who covered the last holiday and who's turn it is. It's all completely fair and reasonable and if you have a good team dynamic you can make it work, but it's definitely more complicated than telling Aunt Edna that of course you'll be home for Christmas.
6) Get paid for it
Whether in flexing the hours for the time spend working a page off hours or by getting paid directly for off hours work. No reason to kill yourself for no additional compensation (and there will be those hellish pages or that automated alarm that goes off hourly starting at 3am).
7) Put the operational burden for supporting a thing in the hands of the people who have the ability to fix it
There should be a cycle of:
Get paged
Root cause
Fix
Post mortem
Deploy fix so that thing never happens again
If you don't have ownership over the thing that's paging you, you're at risk of getting paged all night every night for something you have to go convince other people to take time out of their schedules to fix to solve a problem that they don't feel. Not a great situation.
- gaius 12y agoAt your kid's soccer game? A date night? Planning on doing any of those things? Be prepared to get pulled out at any moment to deal with something that could take hours to resolve Haha true story, I was on-call once and I called a guy in another team because I needed him to do his thing. The call was a little weird. The next day he told me he was driving at the time, phone on speaker balanced on the dash, with the laptop open on the passenger seat, logged in with a 3G dongle...
- bobmagoo 12y agoHa, that sounds about right. I've answered pages from the backseat of my car at a rest stop in the middle of nowhere while my infant looked over my shoulder. Makes for good stories at least. I maintain that the pages answered from team happy hours are the most dangerous.
- gaius 12y agoHe hadn't stopped, had the wheel in one hand, typing into a root shell with the other.
- akg_67 12y ago> Who has to be oncall during Christmas/4th of July/etc? This was the biggest issue on our team. Who is going to cover what holidays? We used to circulate a list of company holidays, birthdays, wedding anniversaries, and "special" days for the whole year at the start of the year to the team so that everyone can prioritize the days they don't want to work. Christmas and Thanksgiving holidays used to be worse as no one wanted to work those holidays. Once our team became more diverse, holiday coverage became little bit easier.