What's up with the web server?

[Tad Carlucci]

I have some concerns about the performance of the KittyCatS.biz web service.

First off, I'll speak to those comments I'm seeing on other threads about posts going missing. The forum software updates the last post time and username whenever a change is made. So, there are times when you'll see it saying so-and-so made the last post at such-a-time but, when you look, that's not the last post and it's not by the named person. That usually means the someone edited (or removed) an earlier post. The clue to this is usually that clicking the Last link from the Forums list will take you somewhere into the midst of the thread, instead of to the very last post.

My issue, though, is with the extreme time it takes for some pages to load.

Recently, I've been looking intensely at Charm's Pedigree and Cattery pages. What I've noticed is these pages can take anywhere from a few second to upwards of a minute to load.

This morning, I've been running some monitoring tools. What I've found is that network latency is NOT the cause. I've been monitoring the time it takes for a response from the various important points:

* The gateway router from my cable company to a backbone. Located in Dallas, TX.

* The gateway router from the backbone to the hosting company. Located in Alexandria, VA (Washington, DC).

* The local router directly connected to KittyCatS.biz. Located near Cleveland, OH.

* The machine hosting KittyCatS.biz. Also in Cleveland, of course.

I did this to ensure that network issues were not the cause of the excessive delays I'm seeing. What I found was about what I expected. Network issues are not the cause. Typical round-trip times were generally around 60 milliseconds (about right for my cable modem). Worst case times were around 200 milliseconds (not bad .. Charm was logging into Second Life to cause those).

So, while network latency might cause a fraction of a second's delay, it cannot account for the seconds-to-minutes delay I've been seeing.

Next, I checked the responsiveness of the main KittyCatS.biz home page. For this, I used some web-based tools. I didn't see much wrong here. The page begins loading almost immediately. I takes about 4 seconds to load, but that's *ALL* because of the two huge images on the main page.

So, I went into my Firefox and started timing how long it took to for Charm's Pedigree Roster page to load.

THAT is where I finally found the wide variation in time. Typically, her 600-or-so cats load in about 2 seconds. I'm *NOT* talking about the page-display time (how long it takes for everything to load and the page to be fully displayed). All I measured was the time it took for the base HTML to be produced and transmitted by the pedigree2.php script.

This leads me to conclude that there is something causing a great deal of slow-down on the web server, and it's only related to the Pedigree and Cattery PHP scripts.

My first guess would be memory page swapping. There's simply not enough physical memory for all the PHP processes and this is causing page swapping.

My next guess would be something having to do with the SQL database. Perhaps it, and not the PHP processes, is page swapping. Or perhaps there are too many concurrent requests for the current configuration.

But, whatever the cause, it's related mainly to the Pedigree and Cattery PHP scripts. Yes, I've noticed occasional problems with other portions of the web server, such as the forum PHP scripts. But that is generally rare, as compared to how often I'm seeing dismal response times from the Pedigree and Cattery.

[Khea Karas]

Total: 370 Cats (? 189 ? 181) | 1913 Boxes (? 957 ? 956)

With the total above my pedigree page can also take a while to load but I supposed that to be because I am located in France, and the amount of cats/boxes I have. It can range from anywhere between 20secs to 1min. I did notice this to increase in loading time over the past 3 months I have to admit. I wonder if this is due to a provider change or something? It also might be because there are more people accessing the server during certain periods of the day. As much as I love the "Build a Cat" tool, I wonder if this also isnt causing added strain on the server being as popular as it is. I would imagine they would have a bandwidth quota and perhaps going over? I have also noticed a huge increase in time with cats creating boxes, and the boxes then showing up and registering with the server (perhaps this a SL issue). These are just my thoughts/questions, I'm no expert so feel free to correct me if I'm wrong.

[Serena Stroikavskoi]

Tad,

Have you taken into consideration the ping time to linden labs for the server side script to obtain all the statistics for the kitties themselvers?

To correctly interpret the response time you also need the response from the in-world server statistics. Which should be good if you have your own in-world to test the rate of response. Also remember Linden was having issues the last 2 days.

[Tad Carlucci]

Linden Labs has nothing to do with the Pedigree or Cattery pages.

---

Oh, I checked access to kittycats.biz from about 50 servers around the world. There was no appreciable difference in timing. Of course, the servers I used are all well-connected, generally near national capitals. So your mileage may vary if your local ISP is not so well connected.

[Tad Carlucci]

Here is a text from Paris using Internet Explorer 8, connection via a DSL.

The only thing to note is that there are two files which don't exist. This slows rendering because the browser retries. You can see, however, that the real slow-down is the two large images.

The number of files being downloaded also is a big slow-down.

And this is *just* the default home page.

I can't use tools like this to test the Pedigree and Cattery because they require a login cookie which I've not found an online tool which can do that. I use Firefox and note the results by hand.

[Serena Stroikavskoi]

(01-17-2013 03:51 PM)Tad Carlucci Wrote:  Linden Labs has nothing to do with the Pedigree or Cattery pages.

---

Oh, I checked access to kittycats.biz from about 50 servers around the world. There was no appreciable difference in timing. Of course, the servers I used are all well-connected, generally near national capitals. So your mileage may vary if your local ISP is not so well connected.

I beg to differ, You are aware that the kittycats is a script on servers hosted by Linden Labs right? And you have seen this correct LSL Portal and this is where the kittycats are dissected, stats gotten, and generally the cattery has to talk to. I believe it's the server hosted about 3200M above the KittyCatS sim. We all know how slow some responses are based on the load.

[Kayleigh McMillan]

Okay I can't read what you just showed, Tad.
I'm just too blond or something but it looks impressive XD
To think this is really what our cats are up to is just like wow.
No wonder they sleep so much.

[Tad Carlucci]

You can differ, but you'd be wrong. The web pages are hosted by InMotion Hosting, Inc. It appears they're a virtual LAMP host .. that is a virtual computer running Linux, Apache, MySQL and PHP. Linden Labs hosts Second Life, which has nothing at all to do with the web pages.

---

I'll expand on that.

Yes, there are scripts running at Linden Labs within the Second Life virtual hosts. Those scripts perform the in-world functions such as selecting the textures and shapes you see on your Second Life Client (or Firestorm or whatever), animating them, etc.

When you rez a cat in-world, it communicates via HTTP (web) requests to the KittyCatS servers, hosted by InMotion. This sends things like where it is (in case they want to track that), WHO owns it, etc. One of the main functions, when you first rez a box, is to ask the KittyCatS servers to generate the genetics and other information for the cat.

The KittyCatS servers include a MySQL (I think, it could be another SQL server) database. On that database is the actual information for every cat.

When you request a Pedigree Roster, the web server runs a PHP script which queries the local MySQL database for the list of cats you own. That list *might* be a bit out of date. For instance, if you sold a cat and the purchaser has never rezzed the cat in-world, no HTTP update from Second Life will have occurred .. so the SQL database will continue to show it on your Pedigree Roster until the purchaser does rez the cat in-world.

The Cattery is a KittyCatS server-side implementation. When you perform an operation, say mating two cats in the Cattery, that operation occurs SOLELY on the KittyCatS servers. Second Life does not participate. Why? Because a cat in the Cattery does NOT EXIST .. AT ALL .. in the Second Life environment. It's simply a collection of information wholly stored on the KittyCatS servers. "Sending a cat to the Cattery" actually involved an HTTP transaction .. informing the KittyCatS servers that the cat is to be completely hosted there .. followed by the cat completely deleting itself from the Second Life environment once it receives confirmation that the KittyCatS servers have taken over control of the cat.

The issues I'm running in to all have to do with the speed at which the Pedigree and Cattery pages are served from the KittyCatS servers. At no point do those pages interact, at all, with any Linden Labs (Second Life) services.

---

The chart I posted is for the main KittyCatS page. "The page" is the very first line. All the other lines are other things (Javascript, images) your web browser will need. The chart shows the time (in ms .. milliseconds .. 1/1000ths of a second) taken to load, interpret, and display, all of the parts needed to view the Home page of the KittyCatS web site.

That very first line, none of the others, are where I'm seeing problems when I'm accessing pages in the Pedigree and Cattery. It's taking a VERY long time between the point where the HTTP request is sent, and when that first line actually begins to send the response back to my web browsers. Typically, for Charms 600-or-so Pedigree, for instance, it takes only a couple of seconds. But far too often it's taking several times that long, often in excess of 10 or 15 seconds, and occasionally as long as a minute. I've investigated to eliminate possible causes on our computers here, and at all points along the Internet, up to the virtual computer actually running the KittyCatS web-based services; and I am satisfied that the issue causing the excessive delays is within those KittyCatS servers. Where in them? I can't tell because I don't have access. But the nature of the problem seems very similar many other resource-starvation issues I've seen over the decades, so I'm guessing that's where the issue lies.

[Serena Stroikavskoi]

I still differ because php, mysql, l2l is all different scripting languages, they still have to interpret and communicate to each other. That's why it's a server side script. It functions directly on the server much like a program. We all know how we complain about those too. When the translator is broke or sick, it slows response time. When you take that into play and the bandwidth between say your cable company and mine along with DNS you have slower response time. Yes the they are on linden lab servers hosted by someone else but the communication between scripting still has to talk to each other in-world as well as the cattery.

When the translator is sick we all see the effect.

[Charmayne Brentley]

(01-17-2013 07:22 PM)Kayleigh McMillan Wrote:  Okay I can't read what you just showed, Tad.
I'm just too blond or something but it looks impressive XD
To think this is really what our cats are up to is just like wow.
No wonder they sleep so much.

He confuses me also. Kayleigh. I nod and say, ok. But I'm clueless. I think he makes lots of pretty charts to make us think the cats are up to something. I am sure they are up there in the cattery playing and up to all kinds of mischief. Oh, wait that's the menagerie. Me smooths my blonde streaked whiskers and shuffled back to bed.