26
Jan
15

Update 2015-01-26

DRIVE RECOVERY STATUS
The recovered disk has been examined and looks in generally positive shape to attempt a full asset recovery.
Melanie from AVN has worked heavily with the recovered asset drive on a process to bring the recovered assets back online.

ASSET SERVER STATUS
In the last OpenSim developers meeting, Melanie from AVN offered to contribute her FSasset service to OpenSimulator for use by OSgrid.
This FSasset service provides similar asset service as the SRAS2 service previously used on OSgrid, as well as additional “large scale grid” features.
She has also provided OSG a new asset server and database design to make the best use of it.
As a result, both of the asset servers were taken down to firmware and rebuilt from scratch.
Both asset servers are back in service and Melanie and OSG are hammering away on them building out the new asset services.

OSGRID RESTART
OSgrid restart is getting closer but there is still no specific date or ETA as we test this entirely “new new” asset service design and associated asset recovery.

OSGRID RESTRUCTURING
Discussions held around consolidating infrequently used plazas on a simulator that can load from a selection of oar files as needed, such as for specific events.
Some experiments on OAR loading, merging, and moving for this capability were successful.
More news once additional planning and testing has been done and final decisions can be made.

SPECIAL THANKS
OSgrid would like to offer thanks to everyone such as Melanie, Justin, Diva, our supporters, and everyone who has gotten behind OSgrid during this catastrophe.
The assistance, the patience, and the good wishes are all very much the rays of light we need to keep pushing forward in an otherwise very awful time.
Thank you all!

Next Update 2015-02-01 or 2015-02-02

19
Jan
15

Update 2015-01-19

DRIVE RECOVERY STATUS
The recovered drive has been received by the datacenter.

ASSET SERVER STATUS
With the databases up and replicating on private interfaces, firewall configurations were enabled.
Failover, Failback, Backup, and Restore procedures drafted and in testing.
Additional regions were brought up on the new asset servers for testing.

OSGRID RESTART
OSgrid restart is getting closer but there is still no specific date or ETA as we test this entirely new asset service.

OSGRID RESTRUCTURING
Discussions held around consolidating infrequently used plazas on a simulator that can load from a selection of oar files as needed, such as for specific events.

12
Jan
15

Update 2015-01-12

DRIVE RECOVERY STATUS
The recovered drive has been shipped from the recovery service to the data center.
No ETA yet on when the data center will receive the recovered drive or have it online for us.

ASSET SERVER STATUS
Both new asset servers have successfully passed the first tests hosting OSgrid regions.
Lbsa Plaza and Sandbox Plaza 2 successfully connected to the new asset servers.
OARs were loaded successfully, and assets replicated to both asset servers as expected.
Dan Banner and I were both able to login to the regions, configure our basic avatars, then hypergrid out to other servers successfully and hypergrid from other servers into the new OSG regions successfully.

OSGRID RESTART
OSgrid restart is getting closer but there is still no specific date or ETA as we test this entirely new asset service.
Once the current testing rounds of failover, failback, and backup/restore are complete to all admins’ satisfaction including finalized checklists and documentation, we may be able to offer a tenative ETA for restoring service.

OSGRID RESTRUCTURING
No discussions held this week while restoring test regions and validating basic grid functions took priority.

06
Jan
15

Update 2015-01-05

DRIVE RECOVERY STATUS
With the holiday breaks behind us, we expect the data center to receive the recovered drive from the recovery service soon, but do not have an ETA yet. We hope to have an ETA or the drive itself before the next update.

ASSET SERVER STATUS
Both new asset servers were fully configured and brought online temporarily working as a master-master replication pair.
This was the first full-scale test of the new database replication, robust xassetservice/hgassetservicece and nginx all together, and it was more successful than we’d hoped.
The database, asset server, and nginx all started together without error, and successfully replicated assets from master to slave in both directions.
This means that asset transactions will be replicated in real-time to the standby asset server, regardless of which server we’re using to serve assets.
The next tests involve loading a test copy of Wright Plaza, testing from master to replica and back to master in both directions, checking all OSgrid operators are happy with failover, failback, and that everyone is satisfied the new asset server design will be able to handle the flood once the gates are opened.
Some technical hurdles remain to be worked out and we’re consulting with core OpenSim developers as needed to remove these as OSgrid comes online using an entire new asset service.

OSGRID RESTART
OSgrid restart is getting closer but there is still no specific date or ETA as we carefully test this entirely new asset service.

OSGRID RESTRUCTURING
Restructuring discussions have begun with an eye to rebalancing the plaza loadout, backups, and the various OSgrid services. At this stage its more inventorying the existing resources and services, and scoping out what to leave along and what should change for 2015 and beyond.

Next Update by 2015-01-11 or 2015-01-12.

29
Dec
14

Update 2014-12-28

We hope you had a happy holidays and a merry Christmas, Boxing Day, or other time of celebration.

Despite the holidays and large case of Real Life Interference, some progress was still made this week.

DRIVE RECOVERY STATUS
We expect the data center to receive the recovered drive once the recovery service and data center staff return from holiday breaks in early January.

ASSET SERVER STATUS
After numerous attack attempts were detected against both asset servers, additional extensive hardening steps have been taken, delaying database and asset server work. In practical terms, almost 200 recommended security configurations on both asset servers were checked and reconfigured as necessary including full patching and verified reboots.

OSGRID RESTART
Two major steps remain to restart OSgrid:
1. Start the database replication
2. Start the asset server

Both of these steps require a number of specific reconfigurations and new tuning from the previous OSgrid configuration, which require a number of deliberate discussions before each step of the implementation.

OSGRID RESTRUCTURING
No discussions on this were taken during the holidays.

We hope you have a safe and happy New Year’s Eve and we’re all looking forward to a great new year for OpenSimulator and OSgrid!

Next Expected Update 2015-Jan-04 or 05.

22
Dec
14

News 2014-12-22

DRIVE RECOVERY STATUS
Recovery service completed drive file conversion and restore to a new disk.
Recovery service has been paid with thanks for their tenacity in dealing with such a large and unique dataset.

Caveats:
Recovery service cannot guarantee OSG can use the data once restored to asset server due to various conversions required to recover data to this point.
OSgrid cannot verify restored database and files can be used for asset service for the same reasons.

Next Steps:
receipt of recovered disk
copy to temporary grid asset server
test restart temp asset server
validate temp asset server
plan next steps based on validation results

ASSET SERVER STATUS

Asset Servers 1 and 2:
Hardware active
RAID configured
OS installed
stock database replaced with higher performance database
OS administrivia like additional loadout and hardening
RAID card battery replaced in asset server 2

Next Steps:
configure database replication
Install new asset service
Begin testing
Additional database tuning

OSGRID RESTART
There is NO estimated time of arrival (ETA) for public re-open.
Test restarts for asset server qualification have begun with initial database configuration and tuning.
That being said, obviously progress is being made to that end.
s soon as we can identify an ETA, we will let you know here.

OSGRID RESTRUCTURING
Due to this crash, everyone agrees changes are necessary to infrastructure and operations.

The restarted OSgrid will include changes to asset servers and plazas to accommodate the needed changes and improve redundancy and reliability.

More news on restructuring changes as these item-by-item discussions reach decisions.

Next Update 2014-12-28 or 29.

15
Dec
14

Recovery News 2014-12-15

RECOVERY
Recovery is complete.
Recovery service cannot guarantee OSG can use the data once restored to asset server due to various conversions required to recover data this far.
OSgrid cannot verify restored data can be used for asset service due to conversions required for recovery.
Pending: payment to recovery service – receipt of recovered disk – copy to temporary grid asset server – test restart temp asset server – validate temp asset server validation – plan next steps based on validation result

ASSET SERVERS
Asset Server 1: Hardware active – RAID configured – OS installed – Database installed
Pending: replication – Install and configure ROBUST asset service

Asset Server 2: Hardware active – RAID configured – OS installed
Pending: Database install – Replication – Install and configure ROBUST asset service

GRID RESTART
There is no ETA for public re-open.
Test restarts for asset server qualification are planned within the month.

RESTRUCTURING
Due to this crash, everyone agrees changes are necessary to infrastructure and operations.
The restarted OSgrid will include changes to asset servers and plazas to accommodate the needed changes and improve redundancy and reliability.
More news on restructuring changes as these item-by-item discussions reach decisions.

Next Update expected: 12-21 or 12-22.




Latest Twitter Update

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 223 other followers

Copyright © 2007-2010 OSGrid, Inc. - All rights reserved, except where noted.

The OSgrid Logo, and the word 'OSgrid' are trademarks of OSGrid, Inc. Usage of these terms elsewhere is allowed under certain conditions.


Follow

Get every new post delivered to your Inbox.

Join 223 other followers