September 2023 January 2023 SpotOn Transact DevOps Engineer Transitioned all environments from KOPS to EKS. Assisted multiple teams with tasks ranging from on-site customer networks issues to deep code dives exploring the origins of slowdowns and outages. Made numerous quality-of-life improvements to infrastructure tooling.
May 2022 June 2018 Square ECOM Operations Engineer Continued all work from Weebly. Took over ownership of Weebly/ECOM internal PKI and performed a full redeployment of all keys and certificates due to a hardware bug, also duplicated the host setup in a second data center and configured for dual-master certificate issuance with fail-over. Worked with engineering teams to facilitate two long-needed PHP upgrades while entirely avoiding downtime. Worked with information and platform security teams to determine and perform the work needed to ensure compliance requirements were met and continued to be met. Pushed for and created a notification service to allow on-server and command-line actions to easily communicate back to engineers via chat messages reducing the amount of manual communication needed, especially during outages and attacks. Overhauled log collection, processing, delivery, and storage to improve stability and throughput. Created a chatbot to ease access to data during outages and attacks, deploy various mitigations, alert to adverse conditions, and, most recently, automatically react to certain issues to prevent an on-call contact. Mentored engineers on everything from how to explain a concept to coding to reviewing to troubleshooting. Created and maintained multiple resource documents for transgender and non-binary people that are currently distributed by HR and internal queer communities. Drove IT policy changes regarding name and username changes. Pushed for a change in language for HR: name/gender & preferred name/gender -> legal name/gender & name/gender.
May 2018 April 2014 Weebly Senior Operations Engineer Worked as part of a six person team in charge of all of the company's non-IT infrastructure. Handling everything from daily maintenance to on-call emergencies to projects large and small. Owned IPv6 adoption from obtaining an allocation to implementing router, firewall, and switch configurations for data centers and offices. Handled the transition from OPSF to IS-IS. Conducted addition of peering at multiple exchanges to a transit-only network. Redesigned user data storage scheme to better distribute load across storage cluster, changed how user metadata is stored and retrieved, and wrote all migration code. Created a plethora of tools and scripts that are used every day by team members and engineers. Worked closely with our legal team to collect information required by warrants and subpoenas. Engaged all kinds of providers, peers, and partners to resolve performance and reliability issues.
April 2014 November 2011 OpenDNS Senior Platform Engineer Responsible for maintaining a 100% overall uptime, managing existing system and network infrastructure (20 POPs and 500 physical nodes), planning and deploying new locations and upgrades, monitoring of all systems, and providing a stable and known platform for engineers to develop, test, distribute, and run services and products. Established new and managed existing network peering relationships. Troubleshot network connectivity issues both internally and externally. Authored numerous internal tools, external services, and documentation to ensure future manageability and enhance customer experience.
November 2011 June 2003 Hurricane Electric Network Engineer Expanded, maintained, and troubleshot a global IP (v4 & v6) network running Brocade routers with Cisco switches. Established and maintained network peering relationships. Drove company to be the largest IPv6 network in the world. Worked with customers and internally to trace network and software issues and abuse, to identify their cause, and remediate or mitigate as needed. Managed approximately 350 Linux hosting servers. Developed internal and external tools. Rewrote core backend components of tunnelbroker.net.