Archive for the 'Disruptions & Downtime' Category

15
Apr
11

OSgrid HyperGrid Asset access fixed

Hello Everyone

Just wanted to let everyone know that today after doing some Hypergrid testing I came to the realization that we had a URL not configured properly on our Robust server, which seemed to be preventing access to your OSgrid assets on remote servers. This has since been resolved and tested and seems to be in good working order again, there are no changes needed on your end, the issue was solely on the back end configuration. We are sorry if this has cause any issues for anyone, if you could please test trying to access your OSgrid inventory while on your personal HG standalones or Grids that would be greatly appreciated. If you would like to provide any feedback please join us on our IRC channel at irc.freenode.net on #osgrid channel, or you can also join our web chat at the following URL : http://www.osgrid.org/index.php/freenode

Thanks everyone.

Michael Emory Cerquoni
OSgrid President

26
Jan
11

ATTENTION : Grid Down Time 1/27/2011 at 12:00pm eastern

Hello everyone,

As I am sure everyone has probably noticed by now that our website user registration and web log in have been disabled for several days now.  I would like to reassure everyone that these were simply growing pains that we have experienced with our database and simply nothing else.   Dave Coyle has spent the last few days reviewing and tweaking a lot of our Database queries and indexes and has converted our inventory tables on Danger grid to innodb, something we had planned to do at the last grid outage but simply ran out of time and decided it was better to put it off until we had the proper amount of time, and I am happy to say that the time is now.  Our testing on danger grid has yielded some very positive results in terms of grid and website performance, unfortunately it requires us to take the grid down for approximately 24 hours to complete, might be less, might be more, we do not want to over promise anything at this point.  So starting at 12:00PM Eastern Standard Time on January 27th 2011 we will be downing the grid to being this process.  We will be providing updates as things progress via twitter, so if you are not already following us on twitter you may want to now @ http://twitter.com/osgrid.   I would like to thank all of our grid testers and users for making these improvements possible.

There is also a new release I will be posting on the website today (OSgrid 0.7.1 (Dev) c472764: 2011-01-26).  This version has several improvements towards avatar appearance and log ins, as well as improved Hypergriding and teleporting.  Some of the changes i would like to specificially point out are in the OpenSim.ini itself, the changes are shown below:

lines 293-305

———————————————-

; Maximum outbound bytes per second for a single scene. This can be used to
; throttle total outbound UDP traffic for a simulator. The default value is
; 0, meaning no throttling at the scene level. The example given here is
; 20 megabits
;
scene_throttle_max_bps = 800000

; Maximum bits per second to send to any single client. This will override
; the user’s viewer preference settings. The default value is 0, meaning no
; aggregate throttling on clients (only per-category throttling). The
; example given here is 1.5 megabits
;
client_throttle_max_bps = 70000

———————————————-

lines 718:729

———————————————-

[PacketPool]
; Enables the experimental packet pool. Yes, we’ve been here before.
RecyclePackets = true;
RecycleDataBlocks = true;

[InterestManagement]
;# {UpdatePrioritizationScheme} {} {Update prioritization scheme?} {BestAvatarResponsiveness Time Distance SimpleAngularDistance FrontBack} BestAvatarResponsiveness
;; This section controls how state updates are prioritized for each client
;; Valid values are BestAvatarResponsiveness, Time, Distance,
;; SimpleAngularDistance, FrontBack
UpdatePrioritizationScheme = BestAvatarResponsiveness
ReprioritizationEnabled = true

———————————————-

These changes should have a profound effect on busier regions on the grid, it enables bandwidth throttles so no one person can suck away all your bandwith, as well as re-enables the previously disabled Update re-prioritization which lets the prims closest to your avatar rez 1st for a better experience.  If you experience any issues with this update as usual please contact us on our web chat for assistance.

Thanks everyone and stay tuned!

Michael Emory Cerquoni

OSgrid Inc President

08
Nov
10

Mandatory Simulator Upgrade 11/09/2010

Hello everyone,

As you might already know we have a major upgrade scheduled for Tuesday Nov 9th at 5pm PST (Nov 10th 01:00:00 UTC) This is a mandatory for all sim operators.  If you are running a simulator on osgrid you will be required to update your regions to the new version (yet to be posted).  I will be posting a new version Tuesday morning so everyone can prepare early if they so choose, but we will not flip the switch on the back end until 5pm (PST).  Once we do upgrade the back end older versions of the OSgrid release will no longer be able to connect, the reason for this is a protocol change to how Avatar appearances are handled.

Quote from Git Change Logs:
“It contains a major interface version bump and will NOT work with earlier grid
services. This is preliminary work that will lead to layers support.”

Some of the other fixes will include, Hair in the center of your screen when huds are attached, Prim Piles at <0,0,0> that crash your viewer when touched, Cross Grid Instant Message Issues, Avatar Appearance at log in, and many other performance improvements!

There will be one new change to OpenSim.ini , which you can actually add now before the upgrade and see some significant improvments as well, the changes are as follows below

find the [ClientStack.LindenUDP] section in your OpenSim.ini

add the following line below inside this section

async_packet_handling = true

here is a description of what this setting does:

“Set this to true to process incoming packets asynchronously. Networking is already separated from packet handling with a queue, so this will only affect whether networking internals such as packet decoding and  acknowledgement accounting are done synchronously or asynchronously Default is true.”

If you find this setting has any ill effects please do let us know right away.

We hope that all the changes will improve the overall experience on OSgrid, however there is still a lot of work ahead to improve many things.  If you are interested in reading all of the patch descriptions you can view the git logs here : http://opensimulator.org/viewgit/?a=summary&p=opensim.

If you have any questions or need guidance please join us on our IRC channel @ irc.freenode.net #osgrid or you can use the website chat here to connect : http://www.osgrid.org/index.php/freenode , you can also follow us at twitter for the fastest way to receive grid updates : http://twitter.com/osgrid.

Michael Emory Cerquoni

OSgrid Inc. President

18
Oct
10

**Grid Upgrade Status Update

Hello Everyone,

I just wanted to let everyone know that the updates are going well, things are taking a bit longer than we expected in converting some of our databases to innodb from myisam.  We wanted to convert all of our databases to innodb, but it looks like its going to just take too long, we did however get about 50% of the tables converted so its a start, unfortunately if we were to convert the inventory table it would have taken at least another 24 hours and we felt that was just too much downtime for this outage.   Things are late now for the OSgrid admins and we need to get some sleep, but we will be back early in the morning for a few more hours of work to bring the website and Robust service online.  We would like to thank everyone for their patience and all the testing and reporting everyone has been doing.   Keep checking back for more updates, we will post some more information soon when we are closer to bringing the grid back online, thanks again everyone.

Michael Emory Cerquoni

OSgrid Inc. President

15
Oct
10

OSgrid 0.7.x Release for pre-setup / inspection

Hello everyone,

I have just created a new release for OSgrid OpenSimulator v0.7.x (0.7.1.cd4d7a7).  I must stress though NO ONE SHOULD BE UPGRADING YET!!!!!  This pre-release is so everyone can prep their regions and get a feel for the upcoming ini changes and differences for their automated upgrade processes.  I have written some instructions for everyone to follow that will hopefully minimize problems during the upgrade process and they are available at the following links:

Plain Text Instructions : http://upgrade.osgrid.org/README.txt

Pretty HTML Instructions : http://upgrade.osgrid.org/README.htm

you can also inspect all the individual components that go into a release here http://upgrade.osgrid.org

Again no one should try to perform an upgrade until we announce its time to pull the trigger, then everyone can move forward, this will likely be sometime Late on Sunday Night or Early Monday Morning (Eastern/US Time) so please keep checking back for updates.  You can also join us on our IRC channel #osgrid at irc.freenode.net  or click here http://upgrade.osgrid.org/chat-help.htm if you have any questions or need guidance.

Thanks everyone

Michael Emory Cerquoni

OSgrid Inc. President

11
Oct
10

a Prelude to the big Upgrade to OpenSimulator v0.7

Hello again everyone

The time is finally here for OSgrid to move forward to version 0.7 of OpenSimulator, something we all have been excited about and waiting for and now the time is here. The plan is that this Sunday, October 17th at 9am US/Eastern (13:00 UTC) we will be taking the grid offline for approximately 24 hours to begin transitioning all of the main grid data not only on to v0.7 but also new hardware for the grid and website services. We ask that everyone be patient during this process as while we are confident that things will and should go smoothly, we can not predict exactly how things will actually go. We ask that all sim operators make sure that they have good solid OAR backups and SQL Dumps of their sims and it might even be a good time for you to make an IAR backup of your inventory if you have access to a sim console. These are all good practices you should be doing on a regular basis anyway, but this is a good week to refresh the practice if you have not been doing it religiously.

Some of the things you should be aware of is that sim configurations will change slightly. We will be sure to post some sample configurations and even a full 0.7 release prior to the grid switching over so everyone can take some time to get a good look at things before the actual switch over occurs. We are aware that there may be some issues in migrating databases during the upgrade process of your simulators, but we will have some working solutions available and we will be providing support via the IRC chat room and Forums during the downtime and as the grid starts to come back online and everyone begins the upgrade process. If you have any questions or concerns about this process please join us on our IRC channel #osgrid at irc.freenode.net or via the chat link on the main website.

There are some very exciting things to look forward to in OpenSimulator v0.7 including Windlight Sharing and the ability to control Windlight Settings with scripting commands. We will also have some preliminary support for media on a prim which allows you to show flash and more complex and interactive HTML websites on prims within your simulator, I have even seen examples of people doing VNC and controlling real computers via their simulator. There are a multitude of bug fixes and simulator improvements that are part of 0.7, you can read the developer release notes here (http://opensimulator.org/wiki/0.7_Release). You should however be very aware that there may be bugs that the OSgrid and OpenSimulator developers are still not aware of, because OSgrid is the largest OpenSimulator Grid and testing is very important part of this grid, v0.7 has not got the proper venting it would normally be getting if OSgrid was running it, so everyone should be prepared for some unexpected things, perhaps bugs and even things not functioning as they did in 0.6.9 releases of OpenSimulator. So I will ask once more that everyone please remain patient for the foreseeable future, this is a very major upgrade release and we will do our best to work with everyone during this phase, but we are only volunteers and there are only so many hours in a day, but we will do our best to make sure no one gets left behind or ignored.

And lastly, the website is going to change drastically, while Elgg has served its purposes for the last year, most of the administrators have grown to be very frustrated with it in terms of being able to easily manage content, users and all its quirks and bugs. We made a decision some time back that because of v0.7′s drastic database changes that would have basiclly forced us to completely refactor Elgg once again we came to the conclusion that it was much easier to just start completley over using a framework that made more sense and was available to everyone so more than just the OSgrid administrators can expand and contribute to. After much searching and contemplating we decided that Simiangrid was the best choice for OSgrid and would bring much more experience to the table in terms of getting stuff done. Just to be clear OSgrid will not be running a Simiangrid back end, we will be staying on the OpenSimulator Robust services, we instead have created a SimianGrid Frontend API to allow it to speak to a Robust based back end, called Zira. Our own Dave Coyle has facilitated most of the work for this API and will be working hard this week to tack on the finish touches and buff out the remaining work. However you will notice that a lot of the functionality we have become accustomed to using with our ELGG front end are not yet available, this does not mean we are completely abandoning those functions, it just means we decided that it was more important that OSgrid get onto 0.7 before we spent many more months working out all new code to replace the lost functionality. We will be retaining all the data from the previous website, some will be transitioned eventually, some will not. We will speak more about this over the coming weeks but for now our focus is getting on to 0.7. So one last time I will remind everyone that patience will be key in this very complex transition, in the end it will all be for the better but there will be many bumps and potholes in the road before we arrive at our destination.

Thank you everyone for making OSgrid the awesome place it has become, without all of you we would never have made it this far.

Michael Emory Cerquoni

OSgrid Inc. President

24
Jul
10

***GRID IS ONLINE***

GRID IS ONLINE – MySQL repairs are complete, no data loss, we applogize for the outage but it was ultimately necessary

20
Jul
10

Grid Back Online

Hi everyone.  OSGrid is back online.  We took advantage of the downtime today to do some grid maintenance we were originally going to schedule for a few weeks from now.  We moved the database backend for the central grid services (except assets) to a new pair of db servers.  Our backend dbs had grown so large, and the disks on the machine serving them were sufficiently slow, that running the backups would effectively take down the inventory server for up to an hour.  So we had disabled the automatic, nightly db backups because they were causing inventory problems for users.  But that meant we didn’t have automatic, nightly db backups anymore.  We had to backup the dbs by hand when an admin was available to babysit the grid servers while it ran and kill the backup if they started causing problems.

The fallout from that is that the most recent inventory backup we have is from June 11.  While it appears most users were unaffected by the inventory problems yesterday, some were.  If your inventory is missing and you have an Inventory ARchive (IAR) you can just load it up in a region you’re in and your inventory will be restored.  If you don’t have an IAR to restore from, we can restore your inventory to as it was the date of the most recent backup, June 11.  We apologize for the inconvenience we know this will cause.  To have your inventory restored from June 11, please send your full avatar name via email to info@osgrid.org.  Depending on the number of people affected this may take us a day or two to complete the restoration for them.

The good news is now that we’ve migrated our db services to a new pair of hosts, we’re able, and have already started, making multiple db backups per day.  This will allow us to minimize impact from any sort of catastrophic failure in the future, and help us to recover more quickly.

It’s not clear at this point what caused the inventory erasure for some users, but we may never really know the answer.  OpenSim is still alpha quality software, and unexpected things can and will happen.  That’s not to say we couldn’t have done a better job with backups; because OSGrid is 100% donation driving we try to be as thrifty as possible, but in this case we should’ve spent the money earlier when we disabled automatic backups to quickly get us to a place where we could turn them back on.  But today we have the benefit of hindsight.

We apologize for the downtime and for the inventory troubles for those affected, thank you for your patience, and thank you for using OSGrid.

-Dave

19
Jul
10

**GRID OFFLINE – UPDATES!

Hello everyone,

First, let me apologize for the grid downtime, for those who are not aware of whats happening, late on Sunday evening it came to our attention that a handful of peoples inventories were reset.   Since we were not sure what was happening we immediately downed the grid in case bad things were still occurring.  We began a database restore for comparison, but because of our current backup server configurations, ie. Software Raid, this machines ability to restore the database is amazingly slow.  On another note some good news, this hardware is scheduled for this week to begin phase 1 of the hardware upgrade to fancy new high-power Hardware raid adapters, sadly this would have already been done a few weeks ago, but unfortunately we ordered the wrong 1u riser boards for our servers and the upgrade had to be rescheduled until we found the proper parts, not so easy!! anyway back to the real issue, because we found out so late about the issue on Sunday night, our volunteer staff tried to stay as long as we possibly could without being so tired that we would likely do more harm than good to the databases.  So I again want to apologize to everyone that we could not stay up all night and fix the servers as quickly as we would like, but you can rest assured that we will be back early in the A.M. on Monday morning trying to make things as right as we can for everyone who is having issues.  Until the database restore is complete we will not have any good explanation as to what really happened, but as soon as we do know i promise you all will know as well, Thank you for your understanding, it is a great pleasure to work with the great citizens of this grid, if you have any questions or would like to report you had inventory issues before the grid went down, please post on the following forum thread : [Grid Downtime Discussion]

Michael Emory Cerquoni (Nebadon Izumi) President, OSgrid Inc.

08
May
10

Upcoming asset service db maintenance

Update: This work has been completed.

~~~

Hello.  This Sunday beginning at 11pm US/Eastern (that’s UTC-4) the OSGrid asset service will be taken offline for database maintenance.  The downtime is expected to be no more than 30 minutes.  The grid will remain up during this time, but item uploads will fail.  No changes will be required by region operators or users.

When the work has been completed I’ll update this post.  I’ll also post to Twitter and in the #osgrid IRC channel on freenode.

Apologies for the inconvenience, and thanks for using OSGrid.

-Dave Coyle

[Comment on this post here.]




Latest Twitter Update

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 120 other followers

Copyright © 2007-2010 OSGrid, Inc. - A California Nonprofit Public Benefit Corporation. All rights reserved, except where noted.

The OSgrid Logo, and the word 'OSgrid' are trademarks of OSGrid, Inc. Usage of these terms elsewhere is allowed under certain conditions.


Follow

Get every new post delivered to your Inbox.

Join 120 other followers