Church IT Podcast Discussions Episode 20 December 6, 2007
Jason
Hey, welcome everybody to Episode 20, the one-year anniversary of the Church IT podcast. You may or may not have read on the blog, we’ve had over 5000 downloads of this thing, averaging 20 or so callers, not counting chat windows. Look forward to seeing what will happen in the future.
Couple of notes, the Sharepoint training opportunity, the pastor had said consultants are not allowed, but we modified that, consultants will get a whopping $1,000 discount off of how much it would normally cost. So go to my blog to get more information, basically $2500 for the week-long training. A huge discount to consultants, we’re trying to work out the registration for that. Do you have anything else on that Jason?
Jason Lee
Right now we are waiting to see if Bill wants us to run the registration. The last email I had from him said they would probably need to run it, I expect to hopefully have an answer from Bill today, so we’ll probably have that registration info by tomorrow.
Jason
So if you know of Church IT Consultants that might want to take advantage of this, let them know.
Ok, today we’ve got Dean and a couple other guys from ACS Technologies and we’re gonna ask them to give us the behind-the-scenes of ACS as far as routers and switches and sans and storage and all that good stuff. So Dean, do you want to introduce who else is on the phone first?
Dean
Sure, we’ve got Josh Wise who was with me at the Roundtable in Kansas City, and also have one of [Time Stamp00:04:38] my other Senior Network Administrators James Munn (sp). James and Josh really keep this place running, they know the ins and out, they are here to answer your questions. I’m just here because you know me.
Hello from James and Josh
Jason
What is your position Dean?
Dean
I’m the IT Director for ACS both on corporate side and on the client side, it’s my responsibility also to make sure all our website servers are up, and access servers are up, we do an on-demand system, anything that is hosted out of our Florence and Charlotte locations, I’m responsible for all the computers, technology, and so forth. James and Josh also split that fence, there are 25 people in IT here, 23 of them are on one side of the fence or the other, either on the client side or the corporate side, but the 3 of us work on both sides, depending on what the problem might be.
Jason
Tell them what ACS Technologies is.
Dean
ACS Technologies is a church management software and solutions provider. We work with churches of all sizes, we originally and for the vast of our history had desktop application products, just in the last 7 or 8 years really started moving everything over to the web front. We’ve got several thousand churches using our web products, and close to 45, 000 churches using our desktop products. We know about desktops and global policies and Internet connections and routers and firewalls, so it’s what we live for. We do it for the churches that don’t have IT staff, for the most part. We are here to support you.
Jason
What I’d like to do is to do an outside to inside look at the technology you use to make ACS run starting with you win connections and offsite data center. Maybe start there.
Dean
We [Time Stamp00:08:34] started out everything hosted from the web perspective we had put in a data center located in Atlanta, Ga. That was a great move for us but it was 5 hours away, it didn’t make sense for us to everything where we couldn’t get to it. So we began building the infrastructure in Florence to support it. Of course the first question from any church who understands the Internet, how many connections to the Internet do you have, what size are the connections, are they redundant, are they bound together to look like one [connection]. We had to build all that and the company we chose to help us do that was F5. It’s been a great solution for us, we are on our second instance of F5, we are getting ready to bring up new ones. We have two main Internet connections into our building, we had Time Warner who was our provider here locally bring us a dark fiber in so they can run up to 100 megabits per second into our building, and Bellsouth, who is now AT&T has another connection into our building, we are on a smart ring with them around the city, so we have 2 connections bound together. We were figuring up yesterday, I think we are running at 65 megabits right now combined. We had Time Warner bring us a second connection.
Josh/James
We had Time Warner because we had so much Internet traffic, we needed a dedicated connection to Internet, so we brought in another 20 meg.
Dean
And all that split out F5. The F5 handles which traffic goes down which connections. It’s a really sweet thing to monitor. Right now we are running a little over 13 megs per second in and out. Today maximum [Time Stamp00:11:27] looks like around 50%. Could be 100s of megs of video or audio or stuff like that. We host all these websites so literally every church, contributions, online giving, things like that. We also have equipment running in Charlotte as well because we needed an offsite location. That’s a little bit about our Internet connections.
Jason
If anybody has a question, just hop it.
Sp
You mentioned live store? It is my understanding there is not a security solution for the data in transit to the live store server, correct? Is that a plain ftp transport to the live store server, that’s why we declined.
Dean
That’s right as of today, funny you mentioned that, literally the last 3 days we have been trouble-shooting our F5 units because the F5s are the reason the secure ftp traffic is not allowed, and we can’t find a solution, which is why we are deploying the new F5 solution to resolve that issue. The code is already in the client to turn on the secure ftp, just right now, our F5s don’t support it.
Jeremie
How many links do you have coming in through that F5 box?
Dean
From the Internet, we have 3 connections coming in. Different speeds.
Jeremie
Are you having good luck with that? What advantages do you get with that instead of just a router and some load-balancing?
James/Josh
Basically with the F5 device, we can take two diverse paths to the Internet and combined them so it’s gonna look like one. The F5 device is smart enough to say when you do a resolution, it’ll pick which path is the best and give you that path. We have the roadrunner connection, if you try to make a connection into our system, when you ask the F5 which path to take, it will pick the one with the most bandwidth or however we set up the load-balancing ratio and it will [Time Stamp00:14:46] then give back the credentials or the IP addresses to come down the correct one.
Jeremie
So does that have to be the authoritative DNS for your whole system?
James/Josh
Not for your entire system, you can delegate just the pieces you want to delegate to the F5 unit and let it handle just the pieces you want to load-balance.
Dean
I know you guys are cost conscience so one of the things the F5 does for us is our Roadrunner bandwidth is a lot cheaper than our AT&T bandwidth. So I can call my Roadrunner rep and say, “move me from 20 by 20 to 30 by 30” and that may only cost $200 more a month, it would cost $6000 more a month. So we can ramp up on the Roadrunner side and give priority to the Roadrunner bandwidth but always have enough on the AT&T side to, if the Roadrunner were to completely fail, we know we’ve got enough to run. We could bring more connections in and have all the bandwidth at different rates.
Jeremie
I think that answers what I was looking for.
Dean
We have multiple F5 units so we are in high availability mode at all times, things like that. It’s been a blessing to have these devices, they work well. You can may a lot of changes on the fly, doesn’t require a lot of reboot. It’s solid equip.
The F5 acts as a firewall because you are only allowing certain ports down certain IP addresses.
James/Josh
Our firewalls are in front of the F5 devices, so we weed that out before we send any traffic to our load-balancing. That way we can kill anything potentially harmful before it gets in.
Sp
So you guys have firewalls in front of your load-balancing?
Dean
Yes.
Jeremie
How much would what you have cost?
Dean
Let’s just say I buy a house about every month. I don’t really know. We’ve probably spent $120,000 [Time Stamp00:18:26] just on F5s.
James/Josh
We also have the same set up in our hosting facility, everything we have here, we pretty much tried to duplicate there, up there we only have one hand-off from our ISP but it’s just there in case.
Jason
Are you using the exact same models of firewalls everywhere?
James/Josh
Core piece is exactly the same models. The reason, the tunnels play a lot nicer when you have like models. Our corporate side piece is split off a little different and engineered a little different, so it does have a smaller class firewall, we’re not load-balancing that.
Dean
We put a lot more energy into our client-facing stuff than we do our corporate because I can explain a little bit easier to my internal folks why their intranet is down than I can to a customer who is trying to make life happen. Firewalls sit in front of the F5, that’s news to me but that’s good. That’s why these guys are smarter than I am. Then of course we split the F5s off, we have a lot of VLANs running here so we can split hosted traffic off and we can split corporate traffic and we have the backbone VLAN as well where we push data across for back-ups.
Sp
How many VLANs?
Dean
About 30. We went down this path for a while where we put a VLAN in for every department, we’ve got 15 departments, support is one.
James/Josh
Our backbone, all our switches and everything communicate, all our firewall and infastructure runs across our backbone VLAN, we have the hosting, we run VoIP and it has it’s own VLAN so we can do quality of service. We’re talking to you on a speaker phone over VoIP, running over all that traffic, network.
Sp
What’s the voice platform?
James/Josh
Avaya. We use Avaya switches for our corporate users and we use the Aspect switch [Time Stamp00:22:11] which is the high availability mission-critical switch for our call center. Aspect was something unknown to me when I first came on board, but it’s the switch Delta and American airlines use, it was a purchase out of our league, too expensive, but back in the late 90s the entire church management industry was suffering, we were all in the dark days, everybody trying to transition to windows, our president said we were going to improve that by fixing our customer service, we’re gonna make sure our customers can get to us when they need us and that we’ve got enough availability to grow, etc. We went out and spent a lot more money, probably $700,00, more than we could afford at the time and that’s been a great switch for us. It’s been in operation 8 year and has not had one minute of unscheduled downtown. It still runs on Windows NT 4, so a little outdated. Come in, we’ll show you the switch.
Jason
What client phones are you running?
Dean
I just got a new phone, one of those 4621 SW, got a backlight, the Avaya is running a 4620 or 4610. We run a hybrid system on our telecom, the main reason is because when we put this system in we did not have the budget to go back and re-cable our existing buildings so we had to run copper for that building and VoIP for our new technology building.
I don’t know if any of you have ever been here other than Tony Dye, Trace has been here, if you’re ever coming down I-95, we’re at the intersection of I-95 and I-20, you’re more than welcome to stop in for a tour, we’ll show you around, let you see all this stuff.
Jason Lee came here but he didn’t have a blog then.
What else you want to hear?
Jason
What kind of switching gear are you running?
Dean
For the most part, everything is HP that has been replaced in the last couple years. James came in [Time Stamp00:26:19] to my office one time and said we could renew our Foundry switches, $16,000 worth of renewals just for support, or he said we could go ahead and buy four new HP with lifetime warranty, so he did the math and the research and it made sense to replace them all.
Jason
Are you sending your guys to HP training?
Dean
No, these guys are sharp. You can stick ‘em on projects and they’ll figure it out. Even the F5, they played with it and figured it out.
On the server infrastructure, from the core network, we’ve got 68 devices that we manage, and that’s switches and routers and the edge stuff, etc. Back-up server, back-up libraries, 7 SANs, 40 switches, Internet connection, APC power, a generator, core router, a couple of Barracudas, wikis, that on our core. Corporate, we run 52 other servers and those are things from the same stuff you guys do, DNS, Exchange, SQL server, file server, print server, all those. 30 devices in our Charlotte facility, 29 running Telecom. That 29 includes the voice mail system, the chat server, the Avaya here in Florence, the Avaya in our Phoenix office, Aspect, a few bricks, the biggest impact on our department is our client side, we run 167 servers right now, that grows by the day. James tells me between physical and virtual since Feb. we brought on over 220 servers, most of those virtuals.
James/Josh
We are not running ESX, just VM ware server. We did some investigative work, the ESX needed to be tied to too much certain equipment especially for the SAN, we’re actually fixing to move over to Xen.
Dean
This is where it could get interesting, I was on the chat the other night with Jason and some others and it came up about Promise Arrays, we’re not on ESX, and we’re going to Xen, it would be interesting to see what you [Time Stamp00:30:46] guys think about that. What are your thoughts on that?
Jeremie
What are you guys using to manage your virtual infrastructure?
James/Josh
That’s why we’re looking at setting up some Xen farms. Right now, it’s manageable but it’s getting to the point where we would be a lot more efficient if we could get all of our servers to 2 management consoles.
Dean
You gotta also remember for us, we’re mission-critical because we serve customers, a lot of what we have is load-balanced so if a virtual goes down, we generally don’t feel it, unlike in our corporate stuff, we don’t have everything clustered or load-balanced so if that server goes down, we do feel the pain. But I’ve got 7 network admins right now and could use another 3 or 4 today. I have an open position for ACS Technology network administrator. It is a challenge to keep up with all this, but this stuff just runs. We don’t have that many problems.
Jason
Talk about what kind of SAN stuff you’re using. A lot of us are interested in the Promise Arrays because of low cost,
Jeremie,
There’s a question on the chat about what router and firewalls their using.
Dean
I’d rather not say on the firewall, for security reasons. But from a router perspective, everything is F5.
James/Josh
The core infrastructure on the network side would be HP routers. You could classify the F5 as a router possibly but it’s more of a load-balancer, traffic is passed in, once you get in, you’re going through HP devices, and we moved from a Big Iron Foundry unit to HP simply because of the price difference on the maintenance plans. We felt like that was a win for us.
Dean
We’ve had great success with the HPs, they’ve been pretty solid. The other priced themselves out on the annual renewals. Sometimes the perception for us is, “oh [Time Stamp00:35:00] you guys are ACS and you’re big and you got 300 employees and you got money,” and that is not necessarily true. We looked at Promise Arrays, EMC, and Equal Logic and all that.
James/Josh
We had Equal Logic come in and do their show and it was sweet, but it was a big difference in price. And we were growing so fast, we needed something easy to manage and the Promise Arrays fit that bill for us. At one point, before we started using the Promise Arrays, we were looking at the Infastore [?], but they didn’t have an iSCSI solution at the time, so you had to buy the SCSI-type SAN and attach it to a head unit and then run iSCSI software on top of that. But with the Promise Array, you buy those, put some drives in, plug them into your iSCSI network and start carving out space.
Jason
Have you had any major component issues?
Dean/James
No. Don’t jinx us. Not even the power supply has gone bad yet. We put the first one in in 2003, we had them in the admin building. There are 7 SANs on the corporate side, 5 that I know of on the client side and at least 1 or 2 in Charlotte, and I think every one of them are Promise Arrays.
Dean
This is where I give James and Josh a lot of credit. These guys took this Promise stuff, which doesn’t have a lot of street credibility on its own, and they pitted it against some of the best out there, some other had more features, but when you talk about through-put and speed, and you might take a performance hit on it, that’s a relative term. If you’re pushing gigabits of data at a time, but what is the sustain rate, what are you really trying to push through. James has taken me through our PRTG, 122 requests right now running on one of our SANs, I just picked one at random. That’s running half of our extend product, virtuals on [Time Stamp00:39:14] top of that Array. The biggest issue we had with Promise is they had some firmware issue with their battery, so once or twice a month the battery alarm will go off, but you silence it and it keeps charging.
James
Other then drives, every now and then we’ll get what they call a time-out error on the drive, usually that’s premonition that the drive is about to fail and what we normally do is just go ahead and yank the drive and put another drive in.
Dean
Disk space is so cheap now.
Jason
That speaks well for Promise, hearing you guys talk about them like that.
Dean
If anybody wants to see any of that, let us know. Good quality products.
James/Josh
Just another piece of information on the Xen technology that we’re looking at, we’ve been testing that over the last few weeks with our Promise Arrays and one of the things we really like about that is that we now have the ability to migrate a live running virtual from one physical box to another physical box as long as it is on the shared SAN, and that shared SAN is on the Promise Arrays, and that worked right out of the box for us, no special drivers, with the Xen piece, we installed it and 10 minutes later we were failing over an XP box -- all running off the Promise Arrays.
Jason
Cool!
Jeremie
A couple folks are asking for your contact information. Can you put that in?
Jason
My guess is that you are going to get a few resumes!
Dean
What else you guys want to hear? Jason has asked about the wiki before.
Jason
Yeah, so you’ve got all these devices, how do you handle change-management. Somebody makes a change or adds a new vm or makes a new VLAN. What does the procedure look like?
Dean
Good question. Remember that James and Josh here split the corporate and client, so they are my contacts, they [Time Stamp00:43:21] are the glue for all of what you just asked. Virtuals don’t get added without those guys knowing about it, VLANs never get added, these are the only two in the building with access to our core switches and routers. So the other guys can’t bring on devices that have to be routed without these guys being involved. So first, this basic first level of ‘no big changes’ without these guys. Then from there, James and Josh have both developed a series of spreadsheets and wiki articles that help our other network admins know what IP schemes to use, what routers to us, etc. that all the network admins are trained on how to bring on new virtuals. There is a lot of talking back and forth. Back in January and February, we got a little too loose on letting people make changes without going through the process and we started stepping on each other. We went through 30 days of hell where one network admin would go in and change an IP setting, another network admin would go back in change it back. I finally had to tell James and Josh to change all the passwords, to slow everybody down to start thinking about what they’re doing. We just keep trying to come up with more efficient ways to make sure people have what they need. We don’t make changes unless teams of people agree to make changes. With all our client-hosted stuff, we have four sets of servers for every product. The development team has a set of servers that they have full control over, they can do anything they want to do with it. And then they have a testing set of servers that when the developers finally get it where they think it should be, they push that to the testing servers. They have full control. Then the third set of servers is the staging. The staging servers we use for IT to become knowledgeable on how to deploy that hosted solution. And the development team has zero rights on that server. So if they are trying to do something where they are writing a temp file to the C drive, we say no, not gonna happen, and we push it back down to development. No one has rights to production, with the exception of the net admin who is responsible for that, and James and Josh. So if you want to kidnap somebody, kidnap James and Josh. They are the ones who make sure everything works. We don’t have a big change request system. I’m not a bureaucratic guy, so I don’t have a lot of sign-offs and things like that, because when the change needs to be made, I want those guys to talk about it and make [Time Stamp00:47:39] the change. As long as the three of them agree, I’m ok with it.
Jason
And that’s getting entered into your wiki?
Dean
Yes. We use the same wiki. You ever heard of screwturn? Fast, powerful, we’ve actually recently really embraces it as a department and put as much as we can out there on the wiki. It’s where we keep all of our contact information for each other as well as our contact info for vendors, all our how to documents, we don’t have fancy naming schemes, if you look at it, you probably wouldn’t understand it unless you had access to the wiki. It’s kinda like a VIN, each number has a purpose. When you’re bringing on 5 servers a week, you gotta do something that makes sense. We keep it straight, very cryptic.
Dean
We use screwturn wiki, we have one for only IT, where only IT personnel have access to it. Josh is a developer so he took screwturn and figured out how to integrate it with AD so using our users and groups on AD, we locked down screwturn to only allow IT people into IT ones, then we have a corporate one that’s corporate wide but it doesn’t get much traffic.
You had asked about seeing how it works and how we’ve got it set up and categories. You really don’t have to categorize it, you just tag it, each article can be tagged, then you can go back and look for only IP related or whatever.
I just pulled up one randomly from Josh. The vm ware guest time runs too fast. The virtuals time runs fast, goes through an entire day in 15 minutes, so he found the answer to that, he threw it out on the wiki page, so we know where to go. It’s kinda of our knowledge base, how-to documents, I use it mostly to look up somebody’s home number when stuff doesn’t work.
At some point when we can share screens or whatever, [Time Stamp00:51:52] I’ll be glad to take you through some of that and let you see how that works.
Jason
That’d be cool. What kind of backup do you guys run?
James/Josh
We’re running Backup Exec.
Dean
We do a lot of disk to disk backups as well, we’ve had our share of issues, we were running ARCserve, when we outgrew the existing backup tape library system, we had to expand it and between all the phone calls to India and 28 other countries, we never could get good resolution. They were telling us to configure it in a way that didn’t make sense. After about 4 days of not getting good backups, this was 2 years ago, we finally had to make a decision to buy, Josh can explain it better.
Josh
