Recent changes
From Inside
Track the most recent changes to the wiki on this page.
List of abbreviations:
- N
- This edit created a new page (also see list of new pages)
- m
- This is a minor edit
- b
- This edit was performed by a bot
- (±123)
- The page size changed by this number of bytes
18 April 2024
|
|
20:53 | ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements 7 changes history +37,804 [Rfulcher (7×)] | |||
|
|
20:53 (cur | prev) +3,036 Rfulcher talk contribs Prep for ==== Amalthea swap out three 10TB for three 24TB Disks ==== to begin tomorrow. | ||||
|
|
20:36 (cur | prev) +7,640 Rfulcher talk contribs Arche HUD * It appears that we have gained additional CPU or System temperatures? Realligning the old/new looks like this: | ||||
|
|
20:24 (cur | prev) +8,289 Rfulcher talk contribs New ==== 20240418 Arche Reformat retired 10TiB Disks for use in Europa or other older Storage Servers ==== | ||||
|
|
18:34 (cur | prev) +6,583 Rfulcher talk contribs New ==== Arche Try more shorter duration rebalancing to see if mfsbrick0210 causes more crashing ==== not quite working as expected, rcs and mfs-data get more endangered than there should be which show up after a restart. | ||||
|
|
16:47 (cur | prev) +1,830 Rfulcher talk contribs lfs-us-isr1 DIFFERENCE=.3182 # mfs-data finally finished scan+purge of mfsbrick0202 | ||||
|
|
10:12 (cur | prev) +2,896 Rfulcher talk contribs :: Thu 18 Apr 2024 09:42:09 AM PDT CS crashed 1125348 Apr 18 09:41:58 arche mfschunkserver[24997]: scanning folder /mfsbrick0210/mfs-data/: 99% (570s) : Apr 18 09:53:25 arche mfschunkserver[28076]: scanning folder /mfsbrick0202/mfs-data/: complete (669s) and finally did not crash! * 2024.04.18 10:11:18 PDT - endangered (26455) / | ||||
|
|
07:02 (cur | prev) +7,530 Rfulcher talk contribs :: Thu 18 Apr 2024 06:55:22 AM PDT CS crashed 1126205 Apr 18 06:55:14 arche mfschunkserver[5445]: scanning folder /mfsbrick0210/mfs-data/: 96% (519s) | ||||
|
|
16:26 | ITP:Act/20151208.23:57/20180706-Migrate Perforce into LoopBack 2 changes history +5,506 [Rfulcher (2×)] | |||
|
|
16:26 (cur | prev) +5,510 Rfulcher talk contribs →20240314 tangent broken s24_sophomoregame_team03 svn team: New transient ==== 20240418 broken sinless_theory svn team ==== commit problem. | ||||
|
|
15:32 (cur | prev) -4 Rfulcher talk contribs →20240314 tangent broken s24_sophomoregame_team03 svn team: drop heading depth | ||||
17 April 2024
|
|
23:35 | ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements 6 changes history +50,100 [Rfulcher (6×)] | |||
|
|
23:35 (cur | prev) +3,169 Rfulcher talk contribs :: Wed 17 Apr 2024 11:29:51 PM PDT CS crashed 1128788 Apr 17 23:29:33 arche mfschunkserver[16989]: scanning folder /mfsbrick0210/mfs-data/: 83% (494s) | ||||
|
|
20:38 (cur | prev) +1,656 Rfulcher talk contribs :: Wed 17 Apr 2024 08:32:16 PM PDT CS crashed 1130165 Apr 17 20:31:59 arche mfschunkserver[18614]: scanning folder /mfsbrick0210/mfs-data/: 76% (409s) | ||||
|
|
19:05 (cur | prev) +9,575 Rfulcher talk contribs :: Wed 17 Apr 2024 07:04:01 PM PDT CS crashed 1130466 Apr 17 19:03:46 arche mfschunkserver[1383]: scanning folder /mfsbrick0210/mfs-data/: 75% (422s) # Tried tuning vfs_cache_pressure but did not speed up brick scanning. | ||||
|
|
13:36 (cur | prev) +5,871 Rfulcher talk contribs :: Wed 17 Apr 2024 01:30:24 PM PDT CS crashed 1132624 Apr 17 13:30:13 arche mfschunkserver[1041]: scanning folder /mfsbrick0210/mfs-data/: 64% (340s) | ||||
|
|
09:51 (cur | prev) +16,720 Rfulcher talk contribs :: Wed 17 Apr 2024 09:46:44 AM PDT CS crashed 1134188 Apr 17 09:46:30 arche mfschunkserver[4871]: scanning folder /mfsbrick0210/mfs-data/: 56% (312s) | ||||
|
|
01:36 (cur | prev) +13,109 Rfulcher talk contribs :: Wed 17 Apr 2024 01:35:15 AM PDT CS crashed 1138373 Apr 17 01:35:06 arche mfschunkserver[21063]: scanning folder /mfsbrick0210/mfs-data/: 34% (182s) | ||||
| m 19:52 | ITTR:Reports/rfulcher/2022.10.21 diffhist 0 Rfulcher talk contribs | ||||
| 19:39 | ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks diffhist +729 Rfulcher talk contribs →202404 Daily RSync 2015-2024 and DigiPen Moodle: : 04/17/24, 03:16 - [it-alerts] cron wwwprod89 distancersync 20240417.0015 20240417.0316 50G:96% 96G:91% 126G:90% 416G:71% 154G:80% MoodleSG2024 589G 154G 80% : 04/17/24, 14:53 - [it-alerts] cron wwwprod27 distancersync 20240417.0015 20240417.1453 374G:93% 111G:95% 575G:81% 329G:88% 497G:86% 159G:88% Moodle2024 1.1T 159G 88% | ||||
| 19:33 | ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03 diffhist +231 Rfulcher talk contribs : 04/17/24, 09:34 - [it-alerts] saturn SyncImapToLizardFS.sh 32%/ 44%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff : 04/16/24, 21:31 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 365G 92% /zfs/imap01Archive04 | ||||
| 16:32 | ITTR Talk:Storage/Bricks diffhist +2,762 Rfulcher talk contribs Arche upgraded from 3.2.0-5-amd64 x86_64 Debian GNU/Linux 7.11 (wheezy) to 8.11 (jessie) to 9.13 (stretch) to 10 (buster) and 5.10.0-0.deb10.26-amd64 x86_64 # More of same recent europa:megas,2 pass2 ST10000VN0004-1ZD101 ZA26JQ8N SATA ReAlSec:1672 | ||||
| 15:58 | ITTR:US Servers/Faculty/UniTime2 diffhist +1,893 Rfulcher talk contribs New ===== 20240417 Need newer java version 17 ===== | ||||
16 April 2024
|
|
21:49 | ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements 4 changes history +38,978 [Rfulcher (4×)] | |||
|
|
21:49 (cur | prev) +3,165 Rfulcher talk contribs Restart loop but log each time it crashes: | ||||
|
|
21:24 (cur | prev) +2,342 Rfulcher talk contribs Is slowly making progress, but still crashing when a mfsbrick0210 duplicate chunk is encountered for unknown reasons. | ||||
|
|
21:06 (cur | prev) +17,170 Rfulcher talk contribs New ==== Arche mfs-data CS is crashing while bringing old bricks back online with reload ==== heading # mfsbrick0210 appears to be the brick causing the crash | ||||
|
|
17:27 (cur | prev) +16,301 Rfulcher talk contribs lfs-us-isr1 DIFFERENCE=.4264 # First ==== Arche try to rebalance until three 24TB disks use matches older Bricks ==== from 55% down to 50% difference # mfs-data is doing Segmentation fault when scanning old bricks, bringing online one at a time. | ||||
| 19:24 | ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03 diffhist +1,035 Rfulcher talk contribs : 04/16/24, 16:14 - [it-alerts] saturn SyncImapToLizardFS.sh 31%/ 44%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff : 04/15/24, 21:23 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 366G 92% /zfs/imap01Archive04 | ||||
| 18:45 | ITP:Act/20160512.20:55/2023 lfs-isr1 XenClient XenBatch Zab00 diffhist +1,041 Rfulcher talk contribs === 20240412 XenBatch Zab00 Cron === .... .... Record each pools email report subject. | ||||
15 April 2024
|
|
22:32 | ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements 3 changes history +12,972 [Rfulcher (3×)] | |||
|
|
22:32 (cur | prev) +5,694 Rfulcher talk contribs Fix that update to Deb10 updated default mfschunkserver binary to v3.0.230 so reverted that back to 2.6.0. Start next 5% rebalance to new bricks that should take ~5ish hours. | ||||
|
|
20:34 (cur | prev) +823 Rfulcher talk contribs Start First loop to reload once one additional percent at any new Brick is consumed. | ||||
|
|
20:16 (cur | prev) +6,455 Rfulcher talk contribs lfs-us-isr1 DIFFERENCE=.4561 # mfs-data Finished # lfs-us-courses Finished. # Start new ==== Arche try to rebalance until three 24TB disks use matches older Bricks ==== heading. | ||||
| 21:29 | ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks diffhist +3,603 Rfulcher talk contribs →202404 Daily RSync 2015-2024 and DigiPen Moodle: fix bad wwwprod89 MoodleSG2023 cache mountpoints were not empty. | ||||
| N 17:47 | ITTR:Backup/Bacula/Audits/US/2024-04-12-Incr diffhist +84,814 Rfulcher talk contribs Incr07CN - 0111.2400 (127440 Sec) - 1363831245981 (1.36 TB) - 0110.5414 (125654 Sec) - DP0180+DP0179 | ||||
| 17:46 | ITTR:Backup/Bacula/Audits diffhist +815 Rfulcher talk contribs Incr07CN - ITTR:Backup/Bacula/Audits/US/2024-04-12-Incr - 0111.2400 (127440 Sec) - 1363831245981 (1.36 TB) - 0110.5414 (125654 Sec) - DP0180+DP0179 | ||||
| 16:28 | ITP:Act/20160512.20:55/2023 lfs-isr1 XenClient XenBatch Zab00 diffhist +2,264 Rfulcher talk contribs : Last backup did manage to finish before the next weeks jobs: # Refine === Speed up slowest US_Colleague_Pool & US_XCPNG82_Pool tasks by initially writing to faster CS === | ||||
