Recovery News 2014-12-15

Recovery is complete.
Recovery service cannot guarantee OSG can use the data once restored to asset server due to various conversions required to recover data this far.
OSgrid cannot verify restored data can be used for asset service due to conversions required for recovery.
Pending: payment to recovery service – receipt of recovered disk – copy to temporary grid asset server – test restart temp asset server – validate temp asset server validation – plan next steps based on validation result

Asset Server 1: Hardware active – RAID configured – OS installed – Database installed
Pending: replication – Install and configure ROBUST asset service

Asset Server 2: Hardware active – RAID configured – OS installed
Pending: Database install – Replication – Install and configure ROBUST asset service

There is no ETA for public re-open.
Test restarts for asset server qualification are planned within the month.

Due to this crash, everyone agrees changes are necessary to infrastructure and operations.
The restarted OSgrid will include changes to asset servers and plazas to accommodate the needed changes and improve redundancy and reliability.
More news on restructuring changes as these item-by-item discussions reach decisions.

Next Update expected: 12-21 or 12-22.


Recovery Update

(from irc, oct 15)

< leTwitch> The news is, there really isn’t much new to tell. Three different organizations spread geographically all over Texas are all struggling with a broken raid volume. We get two steps fwd and one back; be patient. There really isn’t a good way to give an eta on work like this; once we can certify that we have the volume back in a readable form, then we can start pinning down certain tasks and registering progress. Even then, ETAs will be just that *estimated* time to arrival.
< leTwitch>  The real showstopper at this juncture is that the data recovery folks can see the data, but when they ship it to the datacenter and we get it hooked up to the server, we cant. So right now what’s happening is a straight up dump of the paths and directories to a big volume (single partition). The last thing that didn’t work for us was a set of partition copies. The thing we’re doing now is the last trick up the sleeve of the engineer doing the data recovery; if this doesn’t work, we’ll be left with nothing more to do than rebuild the server and start rebuilding from archives.
My understanding is, we dont pay anything if they dont get our stuff back, so if thats how it winds up, we’ll be talking with them about getting back what we have already paid them.
< leTwitch> so if nothing else, the end of the waiting will soon come to pass.
< leTwitch> we’ll be moving forward one way or another.
< leTwitch> Heh, I guess that is news of a sort.
< leTwitch> o/


OSgrid Info


We have new information on the recovery process. The array was not able to be rebuilt from the cloned drives due to logical partition errors so the recovery service has performed a full recovery, restoring everything to a single 6 tb drive that the engineer said should boot and that all the files are intact. The additional costs, less the $2302.56 we have already paid them, comes to $2510. We currently have $3432.88 which would make our budget very tight (we still have hosting costs, etc).

Thank you very much for your patience and support!


Recovery details


The recovery firm has responded with some news, some bad, some good. The initial drive assessment determined that they found sector damage, partition corruption, filesystem corruption, and logical alignment failure. They provided an estimate for file recovery that probably exceeds our means at $4600 and would still require a lot of work to put back into place.

However during subsequent discussions with the engineer it has been suggested that cloning the drives would probably be a more viable solution. The good news is that cloning is less expensive and we would have a working partition, which would help get the grid back online sooner. They will need 4 new drives to accomplish this, and we have contacted our data center about this. We will be able to provide more detailed information, as soon as we receive it.

Thanks again for your patience and understanding.

OSgrid, Inc.


Current status


This post is to notify users and other concerned community that OSgrid has suffered a critical failure in one of the primary drives in our RAID10 array. This is a matter of the gravest technical concern, and while we are working diligently with our datacenter services provider to resolve this issue without data loss, it is imperative that utmost care be directed at this process. This means that it will take time to resolve; even time and the greatest of care are not guarantees that we will come through it without data loss.

Consequently, the grid is offline at present, and will be until we can get this issue resolved. We will do our best to keep everyone up to date on the issue and its resolution as we go forward, but please understand that no news is good news, and that we will have the recovery operation as our first priority until some disposition of the issue has been achieved.

We greatly appreciate your patience and understanding in this matter, and apologize for the jeopardy in which this places all of our content.

Thank you,

OSgrid, Inc.



Unscheduled Maintenance

Aug 18, 2014 Grid is OFFLINE for unscheduled maintenance. Follow us on twitter for updates: https://twitter.com/osgrid

Sorry for the inconvenience, we hope to have this resolved as soon as possible.


Update – Release changes for OSgrid opensimulator.0.7.6.dev.e70c71a

Hello everyone,

I have posted a new version of OSgrid Simulator release on the website today, there are some important changes that you should be aware of before upgrading to make sure you retain 100% functionality.

The main changes are in two files, first is the OpenSim.ini . The following settings need to be moved from [Startup] into their own sections.  (link for the fully updated OpenSim.ini )

OpenSim.ini changes:


    ;# {AllowedClients} {} {Bar (|) separated list of allowed clients} {}
    ;; Bar (|) separated list of viewers which may gain access to the regions.
    ;; One can use a substring of the viewer name to enable only certain
    ;; - "Imprudence" has access
    ;; - "Imprudence 1.3" has access
    ;; - "Imprudence 1.3.1" has no access
    ; AllowedClients =

    ;# {BannedClients} {} {Bar (|) separated list of banned clients} {}
    ;# Bar (|) separated list of viewers which may not gain access to the regions.
    ;; - "Imprudence" has no access
    ;; - "Imprudence 1.3" has no access
    ;; - "Imprudence 1.3.1" has access
    ; BannedClients =
    ;# {permissionmodules} {} {Permission modules to use (may specify multiple modules, separated by comma} {} DefaultPermissionsModule
    ;; Permission modules to use, separated by comma.
    ;; Possible modules are DefaultPermissionsModule, PrimLimitsModule
    permissionmodules = DefaultPermissionsModule

    ;# {serverside_object_permissions} {permissionmodules:DefaultPermissionsModule} {Activate permission handling by the sim?} {true false} true
    ;; These are the parameters for the default permissions module
    ;; If set to false, then, in theory, the server never carries out
    ;; permission checks (allowing anybody to copy
    ;; any item, etc.  This may not yet be implemented uniformally.
    ;; If set to true, then all permissions checks are carried out
    serverside_object_permissions = true

    ;# {allow_grid_gods} {} {Allow grid gods?} {true false} false
    ;; This allows users with a UserLevel of 200 or more to assume god
    ;; powers in the regions in this simulator.
    allow_grid_gods = false

    ;; This allows some control over permissions
    ;; please note that this still doesn't duplicate SL, and is not intended to
    ;# {region_owner_is_god} {} {Allow region owner gods} {true false} true
    ;; Allow region owners to assume god powers in their regions
    region_owner_is_god = true

    ;# {region_manager_is_god} {} {Allow region manager gods} {true false} false
    ;; Allow region managers to assume god powers in regions they manage
    ; region_manager_is_god = false

    ;# {parcel_owner_is_god} {} {Allow parcel owner gods} {true false} true
    ;; Allow parcel owners to assume god powers in their parcels
    ; parcel_owner_is_god = true

    ;# {simple_build_permissions} {} {Allow building in parcel by access list (no groups)} {true false} false
    ;; More control over permissions
    ;; This is definitely not SL!
    ;; Provides a simple control for land owners to give build rights to
    ;; specific avatars in publicly accessible parcels that disallow object
    ;; creation in general.
    ;; Owners specific avatars by adding them to the Access List of the parcel
    ;; without having to use the Groups feature
    ; simple_build_permissions = false
    ;# {GenerateMaptiles} {} {Generate map tiles?} {true false} true
    ;; Map tile options. You can choose to generate normal maptiles or nominate an uploaded texture to
    ;; be the map tile using the MaptileStaticUUID parameter in this section or for individual regions in
    ;; the regions config file(s).  If you do not want to upload map tiles at all, then you will need
    ;; to disable the MapImageServiceModule entirely.
    GenerateMaptiles = true

    ;# {MaptileRefresh} {GenerateMaptiles} {Maptile refresh period?} {} 0
    ;; If desired, a running region can update the map tiles periodically
    ;; to reflect building activity. This names no sense of you don't have
    ;; prims on maptiles. Value is in seconds.
    ; MaptileRefresh = 0

    ;# {MaptileStaticUUID} {} {Asset ID for static map texture} {} 00000000-0000-0000-0000-000000000000
    ;; If not generating maptiles, use this static texture asset ID
    ; MaptileStaticUUID = "00000000-0000-0000-0000-000000000000"

    ;# {TextureOnMapTile} {} {Use terrain textures for map tiles?} {true false} true
    ;; Use terrain texture for maptiles if true, use shaded green if false
    ; TextureOnMapTile = true

    ;# {DrawPrimOnMapTile} {} {Draw prim shapes on map tiles?} {true false} false
    ;; Draw objects on maptile.  This step might take a long time if you've
    ;; got a large number of objects, so you can turn it off here if you'd like.
    ; DrawPrimOnMapTile = true


The following settings need to be moved from the [Startup] in OpenSim.ini to a new section of the GridCommon.ini. (link for fully updated GridCommon.ini)

GridCommon.ini changes


    ; Uncomment the variables in this section only if you are in
    ; Hypergrid configuration. Otherwise, ignore.

    ;# {HomeURI} {Hypergrid} {The Home URL of this world} {}
    ;; If this is a standalone world, this is the address of this instance.
    ;; If this is a grided simulator, this is the address of the external robust server that
    ;; runs the UserAgentsService.
    ;; For example http://myworld.com:9000 or http://myworld.com:8002
    ;; This is a default that can be overwritten in some sections.
    HomeURI = "http://hg.osgrid.org:80/user"

    ;# {GatekeeperURI} {Hypergrid} {The URL of the gatekeeper of this world} {}
    ;; If this is a standalone world, this is the address of this instance.
    ;; If this is a grided simulator, this is the address of the external robust server
    ;; that runs the Gatekeeper service.
    ;; For example http://myworld.com:9000 or http://myworld.com:8002
    ;; This is a default that can be overwritten in some sections.
    GatekeeperURI = "http://hg.osgrid.org:80"


If you have any questions or need help with this update please join us on our IRC chat channel at irc.freenode.net #osgrid or you can use the web chat or on the OSgrid forums here: http://forums.osgrid.org/viewtopic.php?f=24&t=4708

Good luck everyone,


Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 209 other followers

Copyright © 2007-2010 OSGrid, Inc. - All rights reserved, except where noted.

The OSgrid Logo, and the word 'OSgrid' are trademarks of OSGrid, Inc. Usage of these terms elsewhere is allowed under certain conditions.


Get every new post delivered to your Inbox.

Join 209 other followers