Lock the Network Doors and Swallow the Key

There is a rather sensational story on the Drudge Report at this moment about an apparent disgruntled network engineer who granted himself god rights on a network, then locked out everyone else’s administrative rights. He then went to jail rather than divulge his password. It’s the equivalent of locking the door and swallowing the key. City Officials claim that the damage caused by this could be in the millions of dollars.

From the SFGate here:

A disgruntled city computer engineer has virtually commandeered San Francisco’s new multimillion-dollar computer network, altering it to deny access to top administrators even as he sits in jail on $5 million bail, authorities said Monday.

Terry Childs, a 43-year-old computer network administrator who lives in Pittsburg, has been charged with four counts of computer tampering and is scheduled to be arraigned today.

Prosecutors say Childs, who works in the Department of Technology at a base salary of just over $126,000, tampered with the city’s new FiberWAN (Wide Area Network), where records such as officials’ e-mails, city payroll files, confidential law enforcement documents and jail inmates’ bookings are stored.

Childs created a password that granted him exclusive access to the system, authorities said. He initially gave pass codes to police, but they didn’t work. When pressed, Childs refused to divulge the real code even when threatened with arrest, they said.

Details on the technical aspects of this story seem murky, and even a bit suspicious to me. From what I can gather, the city of San Francisco recently consolidated its infrastructure under one common network or portal, and they called it FiberWan.

Supposedly Childs was a disgruntled employee who somehow held onto his job despite attempts to discipline him or fire him. Maybe they didn’t want to fire him because he knew so much about the network and were already fearful of losing his knowledge and expertise?

Unnamed “officials” are quoted fearmongering, claiming that the sysadmin sitting in jail had somehow inserted logic bombs to destroy city documents.

Officials also said they feared that although Childs is in jail, he may have enabled a third party to access the system by telephone or other electronic device and order the destruction of hundreds of thousands of sensitive documents.

And more evidence of his mystic digital foo alleges that Childs was digitally monitoring what HR was doing with his personnel case:

As part of his alleged sabotage, Childs engineered a tracing system to monitor what other administrators were saying and doing related to his personnel case…

I couldn’t find out much about Terry Childs, but he is listed as the network POC for all of the city of San Francisco, according to this ARIN registry page here. This means that he had root access to the city’s routers and backbone.

Which leads me to think that Childs didn’t lock down the network at the workstation/server level. He merely changed the passwords on the routers and may have implemented password bruteforcing lockouts so attempts to bypass his password would result in freezeouts.  If this is the case, it hardly makes Childs the criminal mastermind the media is painting him to be.

But this story should be instructive about whom you trust with god-level access.  It should never be a single person.  And if your infrastructure is so critical that a single point of failure can cripple you, you need to identify and eliminate those choke points.





22 Responses to Lock the Network Doors and Swallow the Key

  1. I like stories like this, especially seeing as I am a network administrator.

    Thanks, and keep 'em coming!

  2. I like stories like this, especially seeing as I am a network administrator.
    Thanks, and keep ‘em coming!

  3. That no one seems(?) to know how to re-gain access suggests that Childs may be the only semi-competent guy there. This is very odd.

    One of the first rules of security is that if you have physical access to the hardware, you can almost certainly subvert any software security. Unless he encrypted the contents of every hard disk (unlikely – especially if the system is running and functional), there is always a secondary path for resetting security when you have physical access.

  4. That no one seems(?) to know how to re-gain access suggests that Childs may be the only semi-competent guy there. This is very odd.

    One of the first rules of security is that if you have physical access to the hardware, you can almost certainly subvert any software security. Unless he encrypted the contents of every hard disk (unlikely – especially if the system is running and functional), there is always a secondary path for resetting security when you have physical access.

  5. I agree with the comments about fearmongering. Presumably "city officials" can provide physical access to servers and routers to other equally intelligent non-criminal masterminds. So the router passwords can be reset in, oh, about 35 secs via the serial port, and the servers can be booted into various recovery modes depending on what sort of operating systems they are running. It's the application level passwords that are likely the difficult ones.

  6. I agree with the comments about fearmongering. Presumably “city officials” can provide physical access to servers and routers to other equally intelligent non-criminal masterminds. So the router passwords can be reset in, oh, about 35 secs via the serial port, and the servers can be booted into various recovery modes depending on what sort of operating systems they are running. It’s the application level passwords that are likely the difficult ones.

  7. Preston and John,

    Thanks for the feedback. So its not just me that thinks some of this sounds fishy?

    Preston, you may be quite right about the competence of the one guy in the Tech department. And the article suggests massive consolidation of the infrastructure, which was likely due to budget constraints. And a tighter budget would likely suggest a small staff.

  8. Preston and John,

    Thanks for the feedback. So its not just me that thinks some of this sounds fishy?

    Preston, you may be quite right about the competence of the one guy in the Tech department. And the article suggests massive consolidation of the infrastructure, which was likely due to budget constraints. And a tighter budget would likely suggest a small staff.

  9. Very fishy. The more I think about it, the more it doesn't stack up. The statement Multi-million dollar implies vendor involvement.

    Network consolidation for the City of San Francisco is not something that a couple of enthusiasts tinkered with over the weekend, and would have had a number of project teams, even if the core IT Department only had a handful of staff.

    Many IT Departments these days are just vendor managers. This story sounds like media sensationalism at its best, fanning the flames of distrust for IT staff who handle sensitive information.

    The first question I ask is where are the policies, controls, and checks to prevent exactly this sort of thing happening? Someone should slap the city officials with a copy of ISO 27001. http://en.wikipedia.org/wiki/ISO/IEC_27001

    This is the point where someone has an epiphany at City Hall and wonders whether or not they should have paid those consultants for a security review.

  10. Very fishy. The more I think about it, the more it doesn't stack up. The statement Multi-million dollar implies vendor involvement.

    Network consolidation for the City of San Francisco is not something that a couple of enthusiasts tinkered with over the weekend, and would have had a number of project teams, even if the core IT Department only had a handful of staff.

    Many IT Departments these days are just vendor managers. This story sounds like media sensationalism at its best, fanning the flames of distrust for IT staff who handle sensitive information.

    The first question I ask is where are the policies, controls, and checks to prevent exactly this sort of thing happening? Someone should slap the city officials with a copy of ISO 27001. http://en.wikipedia.org/wiki/ISO/IEC_27001

    This is the point where someone has an epiphany at City Hall and wonders whether or not they should have paid those consultants for a security review.

  11. Yeah, that thought crossed my mind too. The story says he gave the password to the cops but it didn't work. What if the cops fatfingered the password or couldn't find a special character on the keyboard- and the result is that the sysadmin goes to jail?

  12. Yeah, that thought crossed my mind too. The story says he gave the password to the cops but it didn’t work. What if the cops fatfingered the password or couldn’t find a special character on the keyboard- and the result is that the sysadmin goes to jail?

  13. I am a U.S ex Serviceman, Defending my nation is an oath I have taken. This man/systems administrator, has in my view, committed an act of treason against the government and the authorities.

    Acts of treason are punishable by death. This scum of a traitor should get the death penalty. Cyber terrorist.

  14. I am a U.S ex Serviceman, Defending my nation is an oath I have taken. This man/systems administrator, has in my view, committed an act of treason against the government and the authorities.

    Acts of treason are punishable by death. This scum of a traitor should get the death penalty. Cyber terrorist.

  15. To "Marine", there is more than one possibility. Whether the guy accused is a "traitor" or a patriot is still not clear.

    If you have worked with government agencies – some are excellent, and some are astounding in their lack of competence. The excellent organizations often depend on a very small number of skilled individuals.

    It is possible that Childs is that single skilled individual working within the City of San Francisco government.

    The obvious possibility is that this Childs has gone weird for entirely personal reasons. On a national scale this sort of stuff happens regularly, and could well explain this particular case.

    On the other hand, it is also possible that the "treason" is not Childs, but the folk in City government whose actions with which he did not agree. It is possible that he was asked to do something morally wrong, and he refused.

    Ask system administrators for cases where their bosses asked them to do something morally dubious. You will get a rather a lot of stories.

    This entire story smells funny. Even if Childs went nuts, if the city employed someone competent, the entire problem should have been corrected in a small number of days, That this appears not to be true suggests that Childs was the only halfway-able guy employed by the city. Why did Childs refuse to help the city regain control? One possibility is that Childs went odd.

    This sort of stuff does happen.

    Another possibility is that Childs was told to give administrative access to some political hack's idiot nephew. Ask around – this sort of nonsense occurs far too often.

    As to whether Childs is a traitor, a patriot, or a simple nutter – we do not as yet have anyway to judge.

    … unless you think obeying "authority" is more important than defending the principles on which this country was founded. If blind obedience to authority is your belief, you have more in common with the Nazis than the Founding Fathers.

  16. To “Marine”, there is more than one possibility. Whether the guy accused is a “traitor” or a patriot is still not clear.

    If you have worked with government agencies – some are excellent, and some are astounding in their lack of competence. The excellent organizations often depend on a very small number of skilled individuals.

    It is possible that Childs is that single skilled individual working within the City of San Francisco government.

    The obvious possibility is that this Childs has gone weird for entirely personal reasons. On a national scale this sort of stuff happens regularly, and could well explain this particular case.

    On the other hand, it is also possible that the “treason” is not Childs, but the folk in City government whose actions with which he did not agree. It is possible that he was asked to do something morally wrong, and he refused.

    Ask system administrators for cases where their bosses asked them to do something morally dubious. You will get a rather a lot of stories.

    This entire story smells funny. Even if Childs went nuts, if the city employed someone competent, the entire problem should have been corrected in a small number of days, That this appears not to be true suggests that Childs was the only halfway-able guy employed by the city. Why did Childs refuse to help the city regain control? One possibility is that Childs went odd.

    This sort of stuff does happen.

    Another possibility is that Childs was told to give administrative access to some political hack’s idiot nephew. Ask around – this sort of nonsense occurs far too often.

    As to whether Childs is a traitor, a patriot, or a simple nutter – we do not as yet have anyway to judge.

    … unless you think obeying “authority” is more important than defending the principles on which this country was founded. If blind obedience to authority is your belief, you have more in common with the Nazis than the Founding Fathers.

  17. Just another show for dummies, the media has no idea wtf they talking about nor wtf is going, nor that hole town knows how to regain access from local what a silly storry, infact hillariously retarded. Oh, i forgot its USA! nothing new, moving on.

    I should put this along the other stupid things of the day, much more like a joke then a real story but heh was funny for like 5 seconds.

    thanx for sharing anyway.

  18. Just another show for dummies, the media has no idea wtf they talking about nor wtf is going, nor that hole town knows how to regain access from local what a silly storry, infact hillariously retarded. Oh, i forgot its USA! nothing new, moving on.

    I should put this along the other stupid things of the day, much more like a joke then a real story but heh was funny for like 5 seconds.

    thanx for sharing anyway.