[H] Console recently visited the lovely state of Washington for an exclusive tour of the Xbox Live Operations Center. This will be the first time since the Xbox 360 has launched that anyone has had a look behind the scenes at Microsoftís Xbox Live Operations Center (XOC).
Our host for the tour was Jason Coon, a long time [H]íer and a Operations Program Manager for Xbox Live. The Xbox Live facilities are located away from the main Microsoft campus in a nondescript three story building with no outside markings other than a small Xbox logo on the front door of the lobby.
The lobby is very well laid out with a relaxed feel to it. There is a small waiting area with vending machines that provide free refreshments for visitors. There is a long, stylish reception desk where two lovely young ladies were stationed.
Unfortunately, we were not permitted to take any pictures beyond the lobby area. When we inquired as to why we were unable to photograph the actual operations center, a small team of men in camouflage fatigues repelled in through the windows and confiscated our cameras. O.K. that last part didnít happen but it wouldíve been cool if it did. Apparently there is a ďno cameraĒ policy in effect that applies to everyone, even employees.
The next floor is home to the Xbox Live operations center, various other offices and the Xbox Live servers. The top floor is home to rows upon rows of servers. We are not sure how many but saying there were ďa lotĒ is an understatement.
The Xbox Live Operations Center or ďXOCĒ (pronounced zock) is where all the action is. The large room is literally wall to wall workstations and LCD monitors. There are several monitors attached to the wall at the front of the room that display all kinds of system information. There were also a lot of media carousels at one end of the room, each capable of holding 150 games. From what we saw, there was easily enough storage capacity to hold over 3,000 games.
As our tour came to an end it was obvious that there would be no way to keep track of everything we learned in such a short period of time. Trying to remember who did what and what each department was responsible for would be almost impossible so we turned to our favorite Xbox Live Operations Program Manager for a complete breakdown of the Xbox Live Operations Center.
What follows is the most in-depth coverage of the Xbox Live service to date and the first time anyone has explored the inner workings of the Xbox Live service since the launch of the Xbox 360.
Before we get started, what is in that cool trophy cabinet?
Lots of stuff Things like industry awards (did you know Xbox Live won an Emmy last year? ) plus pieces of hardware that represent manufacturing milestones, such at the first Xbox1 motherboard, the 10,000,000th Xbox controller, and things like the first production Xbox 360 that was signed by the entire team (including me).
So who are you and what do you do?
My name is Jason Coon, my Gamertag is JethroXP. Iím one of four Operations Program Managers on the Xbox Live Team. OPMs as we are called have operational responsibility for various features of Xbox Live. That means we work with the Feature Team (the guys who dream up cool new things for Xbox Live) to ensure that the features they want to implement are operationally supportable, and the Development Team (the guys to write the code to create the features) to ensure that they are writing efficient and sustainable code that behaves well, logs itís activity, and doesnít break anything else, and finally with the Test Team (the guys who test what the devs created) to ensure that the features work as expected and perform well under stress load. We then plan and execute the deployment of those features to the Production service working with our Systems Engineers and our Xbox Operations Center. And finally, we are responsible for ensuring that these features continue to perform well for our users, and so whenever there is a problem, we are called in to drive the investigation and problem resolution.
The features that Iím responsible for are Billing and Marketplace. This covers account creation, subscriptions, MS Points, and content downloads. Whenever you see Press Releases that comment on how many downloads weíve had, or how many users we have, Iím usually the guy providing that data to our Marketing and PR teams. Also, if youíve ever redeemed one of those 25-digit codes for MS Points, an Xbox Live Subscription, a 48 hour Trial Offer, or Downloadable Content, Iím the guy who actually loads all those codes onto our service. Iím also active in the Xbox Live forums where I try to help users who have billing and account trouble.How long have you been with Microsoft?
Over five years. I started in Early 2001 working on Hotmail at Microsoftís Silicon Valley Campus. I transferred to Xbox Live (and to the Redmond, WA area) early in 2004.A typical day is?
Anything but typical I am very much interrupt driven. Iím usually online from home by about 7:30am, checking to see if any problems occurring overnight, or if any hot issues are being escalated. Iím usually in the office by 10:00am and juggling several asks from the Feature, Dev, and Test teams. We rely on email pretty heavily, and on a typical day Iíll receive about 300-400 email messages, and none of that is spam. From my office (and home as well) Iím able to connect to the servers in any of our Test and Development environments, as well as our Production environment. I am logged into various servers in our Production environment virtually all day, every day. Iíve got four Xbox consoles in my office, 2 Xbox1 consoles (one retail, one Dev kit) and 2 Xbox 360 consoles (one retail, one Dev kit). I also have a nice comfy couch in my office so that when Iím ďtestingĒ I can do it in as close approximation to real users as possible. I use the retail kits for validating Production functionality and for troubleshooting problems that users report. I use the Dev Kits for connecting into our Development environments to test how new features are behaving.Typical workstation consists of?
Iíve got my Laptop docked and connected to a 22íí CRT (soon to be a pair of 20íí LCDs) and on a KVM Iíve got two other PCs that I use to connect to our development and Production environments. Iíve got a 20íí TV (not HDTV) and a switch box that I use to connect my two Xbox1 consoles and my two Xbox360 consoles.
Why are you located off campus?
Because we didnít want to make the folks at Office and Windows jealous, I mean, if you are working on serious software would you really want to be sitting next to a bunch of knuckleheads playing games all day? Just Kidding The truth is the Xbox Team grew so fast, there simply wasnít enough room on Campus to house all of us. I hear there might be plans to move us back on campus eventually, but for now our ever expanding team will stay where we are.Is this a 9 to 5 kind of operation?
Not at all, especially not for Xbox Live. We run a service that is in use 24/7. We have an Operations Center that is manned 24 hours a day, and in addition we have many technical folks on call during off hours and weekends. Because we are always working on the next version of Xbox Live we also support all of our development and test environments too. We all do whatever it takes, including working late nights and weekends when necessary.How large are the Xbox Live facilities?
For office space, Xbox as a group has three full multi-floor buildings, and shares part of two other buildings in a corporate office park (we arenít on the main Microsoft Campus) plus we have offices in Mountain View, CA, as well as London and Tokyo. As far as datacenter space, thatís classified.How many people work there?
I think that might be classified too.We learned several interesting acronyms during our trip to the Xbox Live Operation Center like XOC and MOM, what do these acronyms stand for?
XOC is the Xbox Operations Center. It is the Nerve Center of Xbox Live, its staffed 24 hours a day, 7 days a week. They monitor everything about the service, including the network, the servers, and user activity such as how many people are logged on at any given time. They all have Xbox1 and Xbox 360 consoles on their desks, and they have copies of every game ever published for both consoles, so if users are complaining of a specific game problem, they can try and reproduce it so that it is better understood when they escalate the problem to the product team.MOM is the Microsoft Operations Manager. This is a Microsoft built system that allows large IT enterprises to manage many diverse systems. It provides tools to test applications, as well as automated monitoring and alerting. If an application throws an event, or writes to a log, MOM can be configured to monitor it and report when itís behaving beyond specifications. Itís a great toolset, and allows us to manage a much larger environment with fewer people than would ever be possible without it.Let's talk raw numbers...
Generally, raw numbers are classified, but Iíll do what I can.How many overall Live customers are there?
More than four million and growing quite fast every day.How many people are on Live at any given time?
Closely guarded secret, but I can assure you, itís a lot and this number keeps increasing.The most ever?
Just about every day, seriously we donít even celebrate breaking all time records anymore because it keeps happening so often.Daily bandwidth usage?
Many gigabits per second.O.K. then how about the biggest day so far?
38 Gb/s during the influx of people downloading the ďBringing it HomeĒ content of E3 this year.What are the primary functions of the two screens at the front of the room?
The XOC very closely monitors the number of users online at any given time to ensure that everything is running smoothly at all times. They also cycle through other various performance counters for different server types, monitoring load and usage. They even have the ability to dynamically re-allocate some servers if load is increasing to the point that itís degrading performance.
f there is a reported Xbox Live outage, how do you know?
I suppose that all depends on who reports it. Iíve never, not once, seen the XOC caught off guard about a problem. They monitor every aspect of the service so closely that any disruption in service is immediately noticed. If the problem is something that requires my attention, they call me right away, even if itís 2:00am on a Saturday.What other types of information can you call up on the system?
We can pull up all types of information from how many people are playing specific games on Xbox Live to how much free memory is available on our Xbox.com Web servers. The purpose of the system is to keep things running smoothly, so we track anything and everything that helps us achieve that purpose.Why 5 copies of each game?
The XOC has five copies of every game published for both Xbox and Xbox 360 because at times we need multiple people to be signed on to the same game at once. When you walk in and see everyone in the XOC playing Halo2, they are really working, really.So when an issue comes in, how is it handled?
Depends on what it is and how it affects the service. The XOC uses a prioritizing system to track problems. Whenever an issue is first reported, they always take a few minutes to gather more data, assess impact, and validate that the problem is real and not a false positive or momentary glitch. If itís a Priority 1 issue, they are waking people up at home within 10 minutes. Fortunately that very rarely happens. Lower priority issues are quickly addressed in the normal course of the day.How many servers are in use at the Xbox Live network operations center?
Classified, lets call it ďmanyĒ.You said seven copies of Live are running on the servers, what are they and why?
In addition to our Production environment that is used by all of our Xbox Live users, we also maintain seven test and development environments that are basically smaller versions of the real Xbox Live. As I said earlier, we are always working on the next version of Xbox Live, so these other environments are what we use to develop and test new features. This is also where we can practice the deployment steps we will use prior to releasing new features to Production. Youíve heard about our semi-annual service updates, our Spring and Fall releases? We actually practice deploying those updates several times in these test and development environments so that when it comes time to do it for real, weíve already worked out all of the issues and everything runs smoothly.What is Partner Net?
This is one of those seven environments I just mentioned, and is the primary environment used by Xbox partners (1st and 3rd party developers) for testing their new code against our new code.How is Microsoft Game Studios classified?
We treat Microsoft Game Studios just as we would any 3rd party developer. We restrict their access to our facility, and we ensure that we donít share 3rd party data with them.Why?
We highly value our relationships with our 3rd party developers, and we donít want to do anything that even gives the appearance of favoritism to our own in-house game studios. 3rd party developers are our partners, but in regards to MGS, they are competitors as well, and we have a responsibility to protect the intellectual property of all of our partners and ensure that they all are supported equally. It doesnít matter that the folks at MGS wear blue badges just like us, there are some things that we simply wonít let them see or do.
Live is down twice a year? When and why?
As I mentioned, we perform semi-annual service updates. These are scheduled updates that add new and enhanced features. We typically do these in the Spring and the Fall, thus the names ďSpring UpdateĒ and ďFall Update.Ē Our spring update was in May this year, and our Fall update is not far off. The way these work is in three steps. First, we take the service down for typically 24 hours so that we can update every server in the datacenter. Second, we spend several weeks testing the updates with selected consoles (all employees). Third, we make the update available to world, and once that happens everyone connected to Xbox Live will be prompted to download the update the next time they logon to Xbox Live.Does MOM still track activity while Live is down and how is this done?
Yes, even during downtime we have MOM running, mostly because MOM also requires reconfiguring when we update our servers, and itís a good way for the folks in the XOC to monitor that services are being restored as expected when we complete the maintenance and start testing before opening the doors to the public again.Spring and Fall flashes are practiced, how is this done and why?
We practice by deploying the updated code to all of our test environments. Each test environment deployment is like a dress rehearsal for Production. Even very skilled systems engineers get tired during a long deployment, so having done it a number of times, and scripting it out so that each person knows what they are doing and when helps to ensure that it all goes smoothly, and helps us to identify any problems with the process before we get to Production. Just like in sports, you practice, practice, practice so that on game day, there are no doubts and everyone knows exactly what to do.Other than custom Xbox 360s, what are some of the other perks of working for the Xbox Live team?
We do get an occasional retail game for free (we all got Halo2 collectorís editions), lots of t-shirts, hats, game posters and such (I even got a Conker shot glass once ), and all of the Xbox Live Arcade games and MSN Games are free for us. We also get early access to many items such as peripherals and games that are often months away from release and we get the opportunity beta test a lot of this at home. Of course all of that stuff is very cool and fun, but by far I think the biggest perk is really being able to work with cutting edge technology. When you spend your day surrounded by multi-proc (and multi-core) servers, packed with GBs of RAM, sporting multi-chassis U360 SCSI RAID arrays, connected via multi-GB/s links, all running custom code that your team developed, interfacing with custom hardware that your team developed, serving up the premier online gaming experience for more than 4 million people around the world, it almost causes you to lose perspective as to what a ďnormalĒ job might be like.Major Nelson is a Nintendo DS owner, what does Jason game on when he has the chance?
Major Nelson is a gadget freak. Seriously, his office looks like what you might see if a Best Buy delivery truck happened to tip over and spill its contents. I was a diehard PC gamer before joining the Xbox Live team. I was a regular LAN party attendee and organizer, and I was constantly upgrading my PC and tweaking my game configuration files for maximum performance. Now, I game almost exclusively on my Xbox 360. My HD TV offers me higher resolution than my PC (1920x1080 on my TV vs. 1680x1050 on my PC), 5.1 surround sound (I only have 2.1 on my PC), and a far more comfortable place to sit, my couch. Iím not knocking PC gaming, I still prefer a keyboard and mouse for FPS games, but Iíve really become a console convert.Some people like to leave their work at the office, do you take yours home with you? Are you a big gamer?
Yes, and yes. The great thing about the Internet is that I can do 99% of what I do in the office from home. Of course the bad thing about the Internet is that I can do 99% of what I do in the office from home, meaning that Iím rarely unavailable to work. The same is true for many of us. Just when I think Iím the only guy on email at 1:30am on a Saturday, Iíll see two other people I work with respond to an email thread.Just about everyone at Xbox Live is a big gamer, which is part of why we put in so many hours, because we really care about the service. Weíve walked in gamersí shoes, we know what itís like when all you want to do is fire up a game and frag some of your friends, but then something gets in the way and prevents that. We know how frustrating that is, so we all put in far more hours than is asked of us to ensure that the service does run smoothly, and gamers never have to worry about whether or not theyíll be able to get online to play, it just works, every time.Lately Iíve been playing Lego Star Wars II, after having spent a month playing nothing but Dead Rising (46 out of 50 achievements, those 5 and 7 day survival ones are killing me).
We saw several nice cars in the parking lot, including a Porsche and Mercedes, please tell me the Porsche was yours.
Lol, depending on the day you may even see a nice Ferrari in the parking lot too, but sadly, none of those are mine. When I was young and single I drove a Jaguar XJS, V12. Now Iím married (12 years this year) and have a son, so ďpracticalĒ is the key word. The blue Honda Accord out back is mine.How does someone reading this become part of the XBL Team?
You can always search our open positions here. In the Products section scroll all the way to the bottom and select Xbox or Xbox Hardware and then click Search Jobs. Generally the Xbox Live team is look for technically savvy and passionate people. We need good thinkers, who can really wrap their heads around difficult problems, and who have the drive and determination to follow through and develop world class solutions. We need people who are passionate about our product and our service, people who can empathize with our users, and are always thinking of ways to improve the experience. And we need people with a good sense of humor and who like to have fun. We work very hard, and as a result we often need to blow off steam in ďunorthodoxedĒ ways, such as impromptu golf tournaments in the hallways with oversized clubs and wiffle balls, so we need people who donít get pissed when a stray wiffle ball whizzes into their office.
Weíd like to thank Jason and the entire Xbox Live team for taking the time to show us the inner workings of the Xbox Live Operations Center. I, for one, have a new found appreciation for the XBL team and their tireless behind-the-scenes efforts to keep Xbox Live running like a well oiled machine. The next time to you log on to Xbox Live, play a game, download a demo or chat with friends using your Xbox Live Vision camera, you'll know exactly how it all happens.
I didnt know live won an emmy
Latest PSU headlines:
Results 1 to 5 of 5
Inside The Xbox Live Operations Center
My god...Xbox has an operations center?! Talk about movin' on up...
What exactly do they "operate"...from the looks of it, it appears to be nothing more than a collection of original Xbox's that didn't sell last gen.
*sigh*...it appears they put even the most diehard of Xbox fanboys to shame.
Xbox Live operation center
They make sure we get our premium paid service
I wonder what would happen if an employee brought a PS3 in the office... all hell would brake loose on XBOX LIVE!"There is no console war, just kids without enough money"
PSN ID: Aift | Gamertag: Aift
Users Browsing this Thread
There are currently 1 users browsing this thread. (0 members and 1 guests)
By ebony & ivory in forum PlayStation 3Replies: 148Last Post: 05-28-2009, 22:33
By Lefein in forum PlayStation 3Replies: 65Last Post: 11-11-2008, 18:52
By PSKala in forum PlayStation 3Replies: 50Last Post: 03-31-2008, 22:09
By StrikeMaster Ice in forum PlayStation LegacyReplies: 32Last Post: 02-07-2008, 12:27
By StrikeMaster Ice in forum PlayStation LegacyReplies: 16Last Post: 02-07-2008, 01:48