19
Apr
15

OSgrid Logins Re-Opened

The additional asset server maintenance has been completed and the grid logins reopened.

19
Apr
15

OSgrid Logins Closed

OSgrid logins are closed for additional asset service work and set additional cluster parameters.

18
Apr
15

Asset Cluster Maintenance Complete

CLUSTER MAINTENANCE
On April 16th, OSG undertook an emergency cluster maintenance to update core cluster software packages to resolve critical bugs.
INVENTORY AND ASSETS
After the cluster maintenance and restart – many of you began noticing “missing asset” errors when rezzing recent items.
From evidence and investigation – these missing asset messages are not related to the updates applied on the 16th.
It appears that for the last month or so – the asset server was unable to write assets to the database, in a way that did *not* cause an exception and alert the asset server of trouble.
Asset updates appear to have been silently discarded by one or both of the asset clusters during that time.
These silent discards may have been “masked” by the various viewer, region, and nginx caches so that these “ghost assets” still appeared viable… until the asset server was restarted for emergency patching.
CHANGES
While investigating what happened, and trying to determine how this was even possible, several changes have been made to the asset cluster to attempt to prevent a repeat of this type of event, or detect and report it IF it starts to happen again.
In response, OSG has changed the database indexing system, added new logging and exception reporting, and additional database write reporting.
Additional asset write-and-verify tests are being implemented going forward.
At this point, the asset cluster appears to be 100% functioning correctly.
GOING FORWARD
Please examine your viewer and region inventory.
Clear viewer caches and reload them, and delete (or move to another folder) any inventory items that report missing assets.
Regions can use the fcache assets, fcache status, and backup commands to check their region cache and restart your region, watching for new or different asset errors.
Any assets which still appear missing may need to be reloaded from their source, or an IAR/OAR made before or during the time the asset database was silently discarding writes.
FINAL NOTE
We feel your pain with these “missing asset” messages, as the OSG admin inventories and assets were not immune and are experiencing the same issues you are.

We deeply apologize for any inventory or asset issues that you have encountered.

17
Apr
15

Asset Cluster Patches Being Applied

An unplanned emergency maintenance to patch some of the asset server cluster software was started yesterday.

Melanie is applying updates to resolve serious bugs in the cluster code.
During this work, you may experience unexpected error messages during login, region start, or when rezzing items.

An additional note will be posted once the updates are complete and tested.

14
Apr
15

New ini file added to opensim

Hey folks, A new configuration file has been added to OpenSimulator and is included in the new build. Here are some details on how it works:

The new file is called osslEnable.ini and will be located in the config-include folder. By default it is disabled, but can be enabled by un-commenting the include line in the new [OSSL] section of OpenSim.ini

This file provides a way to easily enable OSSL scripting functions in your simulator. you can also edit the path for this file and store it outside of the OpenSim folder so you don’t have to edit it each time you update, or you may not even wish to use it at all.

If anyone has any questions about the use of this new file, give me a shout in-world or in irc and myself (or someone else) will try our best to answer any questions.

dan

08
Apr
15

OSgrid Hardware Maintenance 2015-04-07 to 2015-04-08

LBSA Plaza and a couple sandboxes are offline for server maintenance.

We will bring them back up as soon as it is completed.

While there was no ETA, the data center did say “shortly.”

Thanks to Cuteulala Artis for the initial report of her region connection issues and the related Sandbox region issue.

09
Mar
15

OSgrid Update 2015-03-09

REOPENING PROGRESS
Touch wood, but so far, the reopening of OSgrid has gone extremely well.

There has also been a 30% increase in concurrency since return to service.

SERVICE STATUS
While there were a handful of bumps settling in, the new asset cluster has performed extremely well.

Along the way we have found the occasional bad asset, which were nicknamed “chicken bones” and require an admin to intervene.

But, the admin team has been very quick to find and remove these bad assets as they are encountered.

From investigation these assets were long standing corruption in the asset data which existed long before the RAID crash, which the new asset service has uncovered for us to clear up.

After the assets are rezzed, the problem occurs, and an admin clears the bad asset from disk, the uploaded replacement (good) assets match the asset hash and repair the damage.

VEHICLE CODE AND COMMUNITIES
Improvements in the base OpenSimulator vehicle code have led car, boat, and airplane enthusiasts to begin collecting together in their respective communities.

SANDBOX IMPROVEMENTS
Sandbox Plaza III has been updated to provide water space for boat developers, and NPC support is coming soon.

DONATIONS
OSgrid needs your support!

Please visit the donation page and help keep OSgrid strong!

http://www.osgrid.org/index.php/donate




Latest Twitter Update

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 241 other followers

Copyright © 2007-2010 OSGrid, Inc. - All rights reserved, except where noted.

The OSgrid Logo, and the word 'OSgrid' are trademarks of OSGrid, Inc. Usage of these terms elsewhere is allowed under certain conditions.


Follow

Get every new post delivered to your Inbox.

Join 241 other followers