Recent changes

From Inside

Track the most recent changes to the wiki on this page.

Recent changes options Show last 50 | 100 | 250 | 500 changes in last 1 | 3 | 7 | 14 | 30 days
Hide registered users | Hide anonymous users | Hide my edits | Show bots | Hide minor edits
Show new changes starting from 03:35, 2 May 2024
   
List of abbreviations:
N
This edit created a new page (also see list of new pages)
m
This is a minor edit
b
This edit was performed by a bot
(±123)
The page size changed by this number of bytes

1 May 2024

      21:13  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 2 changes history +19,219‎ [Rfulcher‎ (2×)]
     
21:13 (cur | prev) +10,827‎ Rfulcher talk contribs Rebuild Europa metasas array mfsbrick0400 with old Arche mfsbrick.02.12 mfsbrick.02.17 mfsbrick.02.20 disks is working except for smart data from the ExArche SAS verses originial SATA disks.
     
19:00 (cur | prev) +8,392‎ Rfulcher talk contribs lfs-us-isr1 DIFFERENCE=.1389 # John Watkins Bouncing neptune & pluto & ananke for updates. # Check on more mfsbrick-lfs-us-courses-Apr_25 and still did not find goal=1 orphaned chunks # arche temperature weird 60 value is gone now.
     17:01  ITTR:Backup/Bacula/Audits‎ diffhist +201‎ Rfulcher talk contribs Incr07CP - ITTR:Backup/Bacula/Audits/US/2024-04-26-Incr - 0106.5845 (111525 Sec) - 1460587682808 (1.46 TB) - 0106.1138 (108698 Sec) - DP0177 # ''(DP0178 Resume Error due to ceres upgrade and reboot wrote 664G or ~2000G, re-marked as Full)''
N    16:59  ITTR:Backup/Bacula/Audits/US/2024-04-26-Incr‎ diffhist +84,688‎ Rfulcher talk contribs Incr07CP - 0106.5845 (111525 Sec) - 1460587682808 (1.46 TB) - 0106.1138 (108698 Sec) - DP0177+DP0178 # ''(DP0178 Resume Error due to ceres upgrade and reboot wrote 664G or ~2000G, re-marked as Full)''
     11:37  ITFAQ:Workstation Hardware/Redmond On Campus Specifications‎ diffhist +11‎ Tabatha.tipton talk contribs →‎CLASSROOMS

30 April 2024

     12:31  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎ diffhist +1,629‎ Rfulcher talk contribs Europa mfsbrick0400 marked for removal finished, stopped CS while ACCEPTABLE_DIFFERENCE proceeds.
     12:11  ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks‎ diffhist +851‎ Rfulcher talk contribs →‎202404 Daily RSync 2015-2024 and DigiPen Moodle: : 04/30/24, 02:48 - [it-alerts] cron wwwprod89 distancersync 20240430.0015 20240430.0248 50G:96% 96G:91% 126G:90% 421G:70% 148G:81% MoodleSG2024 596G 148G 81% : 04/30/24, 06:49 - [it-alerts] cron wwwprod27 distancersync 20240429.0015 20240430.0649 422G:92% 111G:95% 575G:81% 328G:88% 553G:84% 66G:96% Moodle2024 1.4T 66G 96% Expanded US +400G

29 April 2024

      22:11  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 2 changes history +14,548‎ [Rfulcher‎ (2×)]
     
22:11 (cur | prev) +13,910‎ Rfulcher talk contribs Europa marked for removal progress: raid2 replication is needed 8633 / 372270 (97.69%)
     
16:51 (cur | prev) +638‎ Rfulcher talk contribs :: Sat Apr 27 09:24:47 PDT 2024 Some jbods replication is needed 179 / 3003480 (100.0%) - 2024.04.29 16:44:59 PDT - amalthea:23975079:182/242TiB:75.03% - arche:23976197:216/271TiB:79.72%
     20:30  ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks‎ diffhist +1,254‎ Rfulcher talk contribs →‎202404 Daily RSync 2015-2024 and DigiPen Moodle: : 04/29/24, 02:44 - [it-alerts] cron wwwprod89 distancersync 20240429.0015 20240429.0244 50G:96% 96G:91% 126G:90% 421G:70% 149G:80% MoodleSG2024 596G 149G 80% : 04/27/24, 02:37 - already running - Ran Long... : 04/28/24, 09:20 - [it-alerts] cron wwwprod27 distancersync 20240427.0015 20240428.0919 422G:92% 111G:95% 575G:81% 328G:88% 539G:85% 83G:95% Moodle2024 1.4T 83G 95%
     20:16  ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03‎ diffhist +842‎ Rfulcher talk contribs : 04/29/24, 11:16 - [it-alerts] saturn SyncImapToLizardFS.sh 33%/ 44%local 21%archive 45%alumni 49%student 66%faculty 29%wanic 75%staff : 04/28/24, 20:58 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 346G 93% /zfs/imap01Archive04

26 April 2024

      19:41  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 3 changes history +3,709‎ [Rfulcher‎ (3×)]
     
19:41 (cur | prev) +485‎ Rfulcher talk contribs Europa Marked for Removal lfs-us-isr DIFFERENCE=.0098 # JBOD mfs-data 91.58%
     
18:11 (cur | prev) +1,714‎ Rfulcher talk contribs Jupiter lfs-us-isr DIFFERENCE=.0098 Finished # JBOD mfs-data 88.12% # lfs-us-rcs,courses,moodle Finished # 20240426 Now that Jupiter:mfsbrick0300(62TiB) is back in service we can mark Europa:mfsbrick0400 for removal, soon.
     
17:51 (cur | prev) +1,510‎ Rfulcher talk contribs More mfsbrick-lfs-us-courses-Apr_25 investigation
     15:19  ITFAQ:Workstation Hardware/Redmond On Campus Specifications‎‎ 5 changes history +72‎ [Tabatha.tipton‎ (5×)]
     
15:19 (cur | prev) +28‎ Tabatha.tipton talk contribs →‎CLASSROOMS
     
08:52 (cur | prev) 0‎ Tabatha.tipton talk contribs
     
08:51 (cur | prev) +113‎ Tabatha.tipton talk contribs
     
08:48 (cur | prev) +176‎ Tabatha.tipton talk contribs
     
08:46 (cur | prev) -245‎ Tabatha.tipton talk contribs →‎CLASSROOMS Tag: Visual edit: Switched

25 April 2024

      20:17  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 2 changes history +12,782‎ [Rfulcher‎ (2×)]
     
20:17 (cur | prev) +12,176‎ Rfulcher talk contribs Jupiter lfs-us-isr DIFFERENCE=.0534 # JBOD mfs-data 78.02% # Trying to track down orphaned jbod1 courses chunk owners.
     
16:36 (cur | prev) +606‎ Rfulcher talk contribs Jupiter lfs-us-isr DIFFERENCE=.0534 # JBOD mfs-data 74.57% # lfs-us-rcs,courses,moodle Finished
     19:02  ITTR Talk:Sympa‎ diffhist +1,154‎ Rfulcher talk contribs Finished cleanup 60G 68% /data/services/sympa
     18:50  ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03‎ diffhist +207‎ Rfulcher talk contribs : 04/25/24, 14:57 - [it-alerts] saturn SyncImapToLizardFS.sh 33%/ 44%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff : 04/24/24, 21:14 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 362G 92% /zfs/imap01Archive04

24 April 2024

      22:09  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 6 changes history +9,948‎ [Rfulcher‎ (6×)]
     
22:09 (cur | prev) +1,145‎ Rfulcher talk contribs * lfs-us-courses is missing 1 chunk on old mfsbrick.00.16 from that set of goal jbod1 that we can not identify # Perhaps bring back online which should let duplicate chunks get purged, then mark it for removal so the list chunk can get coppied elsewhere.
     
21:45 (cur | prev) +293‎ Rfulcher talk contribs lfs-us-courses Finished _REP_LIMIT 1
     
19:14 (cur | prev) +2,390‎ Rfulcher talk contribs An Arche sensor temp1 changed from 32 to 60 along with LED and audible alarm as of Apr 21 08:26:01
     
18:37 (cur | prev) +1,503‎ Rfulcher talk contribs Jupiter lfs-us-isr DIFFERENCE=.0914 # JBOD mfs-data 64.51% # lfs-us-rcs Finished # lfs-us-courses 96.24% # lfs-us-moodle Finished Apr 24 08:18:09
     
18:30 (cur | prev) +4,320‎ Rfulcher talk contribs : Tabatha Tipton - 20240424 11:23 AM - Jacob, the houdini-20.0.547-win64-vc143.exe we have on dragons' den is broken. You download it and attempt to run it and windows gives the "this app cant run on your PC" error. I confirmed by downloading the file from the den and running it on my work laptop and it gives me the error as well. OTRSTicket:3064853 — Helpdesk Ticket (Michelle Wong)
     
17:58 (cur | prev) +297‎ Rfulcher talk contribs * Got email reply from Bruce confirming he downloaded the .zip, he asked for moodle data, but I declined. I forwareded our exchange to: susan.grimm, b.maryott, asuka, gary.oyler, jdeek
     18:52  ITP:Act/20160512.20:55/2024 lfs-isr1 XenClient XenBatch Zab00‎ diffhist +605‎ Rfulcher talk contribs New === 20240423 Periodic SGBounceMetaLoggers === 2961GB Freed up
     18:45  ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks‎ diffhist +1,214‎ Rfulcher talk contribs →‎202404 Daily RSync 2015-2024 and DigiPen Moodle: MoodleSG2024 593G 151G 80% # Moodle2024 1.3T 163G 90%
     18:40  ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03‎ diffhist +207‎ Rfulcher talk contribs : 04/24/24, 13:48 - [it-alerts] saturn SyncImapToLizardFS.sh 32%/ 44%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff : 04/23/24, 21:33 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 364G 92% /zfs/imap01Archive04

23 April 2024

      19:42  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 2 changes history +2,907‎ [Rfulcher‎ (2×)]
     
19:42 (cur | prev) +1,115‎ Rfulcher talk contribs Tangent #3007078 bruce.dellinger Lost Work:
     
18:07 (cur | prev) +1,792‎ Rfulcher talk contribs Jupiter lfs-us-isr DIFFERENCE=.1504 # JBOD mfs-data 51.27% # lfs-us-rcs Finished # lfs-us-courses 68.49% # lfs-us-moodle 87.86%
     18:12  ITTR Talk:Sympa‎‎ 2 changes history +253‎ [Rfulcher‎ (2×)]
     
18:12 (cur | prev) -4,321‎ Rfulcher talk contribs Prune progress to 6hour steps. :: Tue 23 Apr 2024 06:09:54 PM PDT 26G /mnt/LOSTFOUND.old/ # ~50% finished
     
17:52 (cur | prev) +4,574‎ Rfulcher talk contribs sympa1 rsync data/services/sympa/LOSTFOUND.old to archives/sympa progress.
     17:51  ITP:Act/20240319.23:55‎‎ 2 changes history +2,590‎ [Rfulcher‎ (2×)]
     
17:51 (cur | prev) +312‎ Rfulcher talk contribs 4/18/24, 12:41 - 4/23/24, 12:13 - [it] ShoreWare Event Notification: Switch: (116) - x11 Warnings
     
17:48 (cur | prev) +2,278‎ Rfulcher talk contribs 20240423 - Dactyl has been rebooting, Johnny cleaned up network interfaces.
     17:41  ITTR Talk:Storage/Bricks‎ diffhist +448‎ Rfulcher talk contribs europa:megas,2 pass2 ST10000VN0004-1ZD101 ZA26JQ8N SATA ReAlSec:2040
     17:33  ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03‎ diffhist +207‎ Rfulcher talk contribs : 04/23/24, 15:10 - [it-alerts] saturn SyncImapToLizardFS.sh 32%/ 44%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff : 04/22/24, 21:10 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 363G 92% /zfs/imap01Archive04
     17:30  ITTR:Backup/Bacula/Audits‎ diffhist -2‎ Rfulcher talk contribs Cleanup up some Container 208108 notes to match the monthly emails
     17:20  ITP:Act/20150625.02:30/202101-Prometheus sm1083224 Blades ARP Metalogger‎ diffhist +10,142‎ Rfulcher talk contribs * Last was #20240423 metaloggers SGBounceMetaLoggers '''2961GB Freed up''' (307+1059+531+1064=2961GB)
 N    16:38  ITP:Act/20160512.20:55/2024 lfs-isr1 XenClient XenBatch Zab00‎‎ 2 changes history +49,119‎ [Rfulcher‎ (2×)]
     
16:38 (cur | prev) +117‎ Rfulcher talk contribs === 2025 Recording in new article === TODO: * See ITP:Act/20160512.20:55/2025 lfs-isr1 XenClient XenBatch Zab00
N    
16:37 (cur | prev) +49,002‎ Rfulcher talk contribs Moved from ITP:Act/20160512.20:55/2023 lfs-isr1 XenClient XenBatch Zab00 into new article for new year.
     16:38  ITP:Act/20160512.20:55/2023 lfs-isr1 XenClient XenBatch Zab00‎‎ 2 changes history -43,884‎ [Rfulcher‎ (2×)]
     
16:38 (cur | prev) -47,613‎ Rfulcher talk contribs Moved to ITP:Act/20160512.20:55/2024 lfs-isr1 XenClient XenBatch Zab00 into new article for new year.
     
16:31 (cur | prev) +3,729‎ Rfulcher talk contribs === 20240419 XenBatch Zab01 Cron === .... .... Record each pools email report subject.

22 April 2024

      19:28  ITTR:Backup/Bacula/Audits‎‎ 2 changes history +1,621‎ [Rfulcher‎ (2×)]
     
19:28 (cur | prev) +146‎ Rfulcher talk contribs * Container 208109 - Incr07CC-Incr07CQ (20240125~20240503) DP0192-DP0177
     
19:17 (cur | prev) +1,475‎ Rfulcher talk contribs Incr07CO - ITTR:Backup/Bacula/Audits/US/2024-04-19-Incr - 0118.5729 (154649 Sec) - 1678644222342 (1.68 TB) - 0118.2334 (152614 Sec) - DP0179+DP0178
N    19:05  ITTR:Backup/Bacula/Audits/US/2024-04-19-Incr‎ diffhist +84,982‎ Rfulcher talk contribs Incr07CO - 0118.5729 (154649 Sec) - 1678644222342 (1.68 TB) - 0118.2334 (152614 Sec) - DP0179+DP0178
      18:37  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 3 changes history +8,363‎ [Rfulcher‎ (3×)]
     
18:37 (cur | prev) +378‎ Rfulcher talk contribs lfs-us-courses7 _REP_LIMIT =2 Expect ~16MBps
     
18:23 (cur | prev) +2,163‎ Rfulcher talk contribs Jupiter lfs-us-isr DIFFERENCE=.1960 # JBOD mfs-data 38.19% (fixed a missing chunk in fg300.log-20240415-2215.gz) # lfs-us-rcs 88.68% # lfs-us-courses 41.28% # lfs-us-moodle 67.45%
     
16:27 (cur | prev) +5,822‎ Rfulcher talk contribs Jupiter lfs-us-isr DIFFERENCE=.2061 # JBOD mfs-data 34.86% # lfs-us-rcs 81.10% # lfs-us-courses 37.73% # lfs-us-moodle 62.24% # lfs-data-ssd Finished Apr 22 07:59:19
     18:04  ITTR Talk:Sympa‎ diffhist +835‎ Rfulcher talk contribs New === 20240422 Stash some data sympa/LOSTFOUND.old to network archives ===
     17:03  ITTR Talk:Storage/Bricks‎ diffhist +2,212‎ Rfulcher talk contribs europa:megas,2 pass2 ST10000VN0004-1ZD101 ZA26JQ8N SATA ReAlSec:2008
     16:53  ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03‎ diffhist +871‎ Rfulcher talk contribs : 04/22/24, 13:47 - [it-alerts] saturn SyncImapToLizardFS.sh 32%/ 44%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff : 04/21/24, 20:59 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 361G 92% /zfs/imap01Archive04

20 April 2024

     01:08  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎ diffhist +753‎ Rfulcher talk contribs DIFFERENCE=.2423 (2.570%) (5.930%) (2.690%) (6.120%) (9.930%)
     01:01  ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks‎ diffhist +457‎ Rfulcher talk contribs →‎202404 Daily RSync 2015-2024 and DigiPen Moodle: : 04/19/24, Ran Long... : 04/20/24, 00:57 - [it-alerts] cron wwwprod27 distancersync 20240419.0015 20240420.0057 374G:93% 111G:95% 575G:81% 328G:88% 498G:86% 50G:97% Moodle2024 1.3T 50G 97% : <code>Sat Apr 20 00:57:23 root@wwwprod27:~# NewLoop 1400 /lfs-us-isr1/LoopBacks/moodle/2024.zfs # 1200 1288490188800</code> +200GB Moodle2024 1.3T 1.3T 50G 97%

19 April 2024

     20:41  ITP:Act/20130612.00:19/2020-Sideloaded Arcade Games‎ diffhist +1,546‎ Rfulcher talk contribs 20240414 Kilonova_Setup.exe
     20:10  ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks‎ diffhist +649‎ Rfulcher talk contribs →‎202404 Daily RSync 2015-2024 and DigiPen Moodle: : 04/19/24, 02:41 - [it-alerts] cron wwwprod89 distancersync 20240419.0015 20240419.0241 50G:96% 96G:91% 126G:90% 417G:71% 154G:80% MoodleSG2024 590G 154G 80% : 04/18/24, None :: Fri Apr 19 20:09:06 root@wwwprod27:~# Moodle2024 1.3T 1.2T 64G 96% # Will need to expand soon.
      20:03  ITTR Talk:Storage/Bricks‎‎ 3 changes history +1,931‎ [Rfulcher‎ (3×)]
     
20:03 (cur | prev) +400‎ Rfulcher talk contribs * 2024-04-19T20:00:04 ::: Amalthea New Disks :: mfsbrick0010 /dev/da5 ST24000NM007H ZYD04AKL :: mfsbrick0011 /dev/da8 ST24000NM007H ZYD057T6 :: mfsbrick0016 /dev/da9 ST24000NM007H ZYD06NSL
     
18:03 (cur | prev) +193‎ Rfulcher talk contribs * 2024-04-19T18:00:04 ::: FreeBSD amalthea.digipen.edu 13.2-RELEASE-p4 upgrade to 13.3-RELEASE-p1 amd64 ::: mfsbrick0015 /dev/da22 became /dev/da21 ::: mfsbrick0019 /dev/da23 became /dev/da22
     
18:00 (cur | prev) +1,338‎ Rfulcher talk contribs More minor pluto:3ware,3 ST4000DM000-1F2168 Z300JVLK and europa:megas,2 pass2 ST10000VN0004-1ZD101 ZA26JQ8N ReAlSec:1768
     19:59  ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03‎ diffhist +414‎ Rfulcher talk contribs : 04/19/24, 14:16 - [it-alerts] saturn SyncImapToLizardFS.sh 32%/ 44%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff : 04/18/24, 21:21 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 366G 92% /zfs/imap01Archive04
      19:57  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 3 changes history +9,419‎ [Rfulcher‎ (3×)]
     
19:57 (cur | prev) +694‎ Rfulcher talk contribs Would like to take these last three JBODfront Slots 21,22,23 offline at some point, change goal to move data elsewhere, if we have ~31+TiB Available at Amalthea:
     
19:39 (cur | prev) +6,962‎ Rfulcher talk contribs lfs-us-isr1 DIFFERENCE=.2548 # mfs-data 3000682 (.1000%) # rcs 1663084 (.2300%) # courses 552725 (.0300%) # moodle 382539 (.9600%) # lfsx202 157529 (.1200%) # Three New Amalthea 24TiB Bricks swapped into service.
     
18:05 (cur | prev) +1,763‎ Rfulcher talk contribs lfs-us-isr1 DIFFERENCE=.2672 # : John Watkins - 4:47 PM - amalthea needs a reboot to upgrade to 13.3
     14:52  ITP:Act/20151208.23:57/20180706-Migrate Perforce into LoopBack‎ diffhist +1,860‎ Rfulcher talk contribs →‎2024 svn errors: * According to trac top is rev1585 * Dumped revision 1582. (Was the top rev when the dump was started.) :: real 383m57.768s :: user 26m16.132s :: sys 8m27.260s

18 April 2024

      20:53  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 7 changes history +37,804‎ [Rfulcher‎ (7×)]
     
20:53 (cur | prev) +3,036‎ Rfulcher talk contribs Prep for ==== Amalthea swap out three 10TB for three 24TB Disks ==== to begin tomorrow.
     
20:36 (cur | prev) +7,640‎ Rfulcher talk contribs Arche HUD * It appears that we have gained additional CPU or System temperatures? Realligning the old/new looks like this:
     
20:24 (cur | prev) +8,289‎ Rfulcher talk contribs New ==== 20240418 Arche Reformat retired 10TiB Disks for use in Europa or other older Storage Servers ====
     
18:34 (cur | prev) +6,583‎ Rfulcher talk contribs New ==== Arche Try more shorter duration rebalancing to see if mfsbrick0210 causes more crashing ==== not quite working as expected, rcs and mfs-data get more endangered than there should be which show up after a restart.
     
16:47 (cur | prev) +1,830‎ Rfulcher talk contribs lfs-us-isr1 DIFFERENCE=.3182 # mfs-data finally finished scan+purge of mfsbrick0202
     
10:12 (cur | prev) +2,896‎ Rfulcher talk contribs :: Thu 18 Apr 2024 09:42:09 AM PDT CS crashed 1125348 Apr 18 09:41:58 arche mfschunkserver[24997]: scanning folder /mfsbrick0210/mfs-data/: 99% (570s) : Apr 18 09:53:25 arche mfschunkserver[28076]: scanning folder /mfsbrick0202/mfs-data/: complete (669s) and finally did not crash! * 2024.04.18 10:11:18 PDT - endangered (26455) /
     
07:02 (cur | prev) +7,530‎ Rfulcher talk contribs :: Thu 18 Apr 2024 06:55:22 AM PDT CS crashed 1126205 Apr 18 06:55:14 arche mfschunkserver[5445]: scanning folder /mfsbrick0210/mfs-data/: 96% (519s)
     16:26  ITP:Act/20151208.23:57/20180706-Migrate Perforce into LoopBack‎‎ 2 changes history +5,506‎ [Rfulcher‎ (2×)]
     
16:26 (cur | prev) +5,510‎ Rfulcher talk contribs →‎20240314 tangent broken s24_sophomoregame_team03 svn team: New transient ==== 20240418 broken sinless_theory svn team ==== commit problem.
     
15:32 (cur | prev) -4‎ Rfulcher talk contribs →‎20240314 tangent broken s24_sophomoregame_team03 svn team: drop heading depth

17 April 2024

      23:35  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 6 changes history +50,100‎ [Rfulcher‎ (6×)]
     
23:35 (cur | prev) +3,169‎ Rfulcher talk contribs :: Wed 17 Apr 2024 11:29:51 PM PDT CS crashed 1128788 Apr 17 23:29:33 arche mfschunkserver[16989]: scanning folder /mfsbrick0210/mfs-data/: 83% (494s)
     
20:38 (cur | prev) +1,656‎ Rfulcher talk contribs :: Wed 17 Apr 2024 08:32:16 PM PDT CS crashed 1130165 Apr 17 20:31:59 arche mfschunkserver[18614]: scanning folder /mfsbrick0210/mfs-data/: 76% (409s)
     
19:05 (cur | prev) +9,575‎ Rfulcher talk contribs :: Wed 17 Apr 2024 07:04:01 PM PDT CS crashed 1130466 Apr 17 19:03:46 arche mfschunkserver[1383]: scanning folder /mfsbrick0210/mfs-data/: 75% (422s) # Tried tuning vfs_cache_pressure but did not speed up brick scanning.
     
13:36 (cur | prev) +5,871‎ Rfulcher talk contribs :: Wed 17 Apr 2024 01:30:24 PM PDT CS crashed 1132624 Apr 17 13:30:13 arche mfschunkserver[1041]: scanning folder /mfsbrick0210/mfs-data/: 64% (340s)
     
09:51 (cur | prev) +16,720‎ Rfulcher talk contribs :: Wed 17 Apr 2024 09:46:44 AM PDT CS crashed 1134188 Apr 17 09:46:30 arche mfschunkserver[4871]: scanning folder /mfsbrick0210/mfs-data/: 56% (312s)
     
01:36 (cur | prev) +13,109‎ Rfulcher talk contribs :: Wed 17 Apr 2024 01:35:15 AM PDT CS crashed 1138373 Apr 17 01:35:06 arche mfschunkserver[21063]: scanning folder /mfsbrick0210/mfs-data/: 34% (182s)
 m   19:52  ITTR:Reports/rfulcher/2022.10.21‎ diffhist 0‎ Rfulcher talk contribs
     19:39  ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks‎ diffhist +729‎ Rfulcher talk contribs →‎202404 Daily RSync 2015-2024 and DigiPen Moodle: : 04/17/24, 03:16 - [it-alerts] cron wwwprod89 distancersync 20240417.0015 20240417.0316 50G:96% 96G:91% 126G:90% 416G:71% 154G:80% MoodleSG2024 589G 154G 80% : 04/17/24, 14:53 - [it-alerts] cron wwwprod27 distancersync 20240417.0015 20240417.1453 374G:93% 111G:95% 575G:81% 329G:88% 497G:86% 159G:88% Moodle2024 1.1T 159G 88%
     19:33  ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03‎ diffhist +231‎ Rfulcher talk contribs : 04/17/24, 09:34 - [it-alerts] saturn SyncImapToLizardFS.sh 32%/ 44%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff : 04/16/24, 21:31 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 365G 92% /zfs/imap01Archive04
     16:32  ITTR Talk:Storage/Bricks‎ diffhist +2,762‎ Rfulcher talk contribs Arche upgraded from 3.2.0-5-amd64 x86_64 Debian GNU/Linux 7.11 (wheezy) to 8.11 (jessie) to 9.13 (stretch) to 10 (buster) and 5.10.0-0.deb10.26-amd64 x86_64 # More of same recent europa:megas,2 pass2 ST10000VN0004-1ZD101 ZA26JQ8N SATA ReAlSec:1672
     15:58  ITTR:US Servers/Faculty/UniTime2‎ diffhist +1,893‎ Rfulcher talk contribs New ===== 20240417 Need newer java version 17 =====

16 April 2024

      21:49  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 4 changes history +38,978‎ [Rfulcher‎ (4×)]
     
21:49 (cur | prev) +3,165‎ Rfulcher talk contribs Restart loop but log each time it crashes:
     
21:24 (cur | prev) +2,342‎ Rfulcher talk contribs Is slowly making progress, but still crashing when a mfsbrick0210 duplicate chunk is encountered for unknown reasons.
     
21:06 (cur | prev) +17,170‎ Rfulcher talk contribs New ==== Arche mfs-data CS is crashing while bringing old bricks back online with reload ==== heading # mfsbrick0210 appears to be the brick causing the crash
     
17:27 (cur | prev) +16,301‎ Rfulcher talk contribs lfs-us-isr1 DIFFERENCE=.4264 # First ==== Arche try to rebalance until three 24TB disks use matches older Bricks ==== from 55% down to 50% difference # mfs-data is doing Segmentation fault when scanning old bricks, bringing online one at a time.
     19:24  ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03‎ diffhist +1,035‎ Rfulcher talk contribs : 04/16/24, 16:14 - [it-alerts] saturn SyncImapToLizardFS.sh 31%/ 44%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff : 04/15/24, 21:23 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 366G 92% /zfs/imap01Archive04
     18:45  ITP:Act/20160512.20:55/2023 lfs-isr1 XenClient XenBatch Zab00‎ diffhist +1,041‎ Rfulcher talk contribs === 20240412 XenBatch Zab00 Cron === .... .... Record each pools email report subject.

15 April 2024

      22:32  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 3 changes history +12,972‎ [Rfulcher‎ (3×)]
     
22:32 (cur | prev) +5,694‎ Rfulcher talk contribs Fix that update to Deb10 updated default mfschunkserver binary to v3.0.230 so reverted that back to 2.6.0. Start next 5% rebalance to new bricks that should take ~5ish hours.
     
20:34 (cur | prev) +823‎ Rfulcher talk contribs Start First loop to reload once one additional percent at any new Brick is consumed.
     
20:16 (cur | prev) +6,455‎ Rfulcher talk contribs lfs-us-isr1 DIFFERENCE=.4561 # mfs-data Finished # lfs-us-courses Finished. # Start new ==== Arche try to rebalance until three 24TB disks use matches older Bricks ==== heading.
     21:29  ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks‎ diffhist +3,603‎ Rfulcher talk contribs →‎202404 Daily RSync 2015-2024 and DigiPen Moodle: fix bad wwwprod89 MoodleSG2023 cache mountpoints were not empty.
N    17:47  ITTR:Backup/Bacula/Audits/US/2024-04-12-Incr‎ diffhist +84,814‎ Rfulcher talk contribs Incr07CN - 0111.2400 (127440 Sec) - 1363831245981 (1.36 TB) - 0110.5414 (125654 Sec) - DP0180+DP0179
     17:46  ITTR:Backup/Bacula/Audits‎ diffhist +815‎ Rfulcher talk contribs Incr07CN - ITTR:Backup/Bacula/Audits/US/2024-04-12-Incr - 0111.2400 (127440 Sec) - 1363831245981 (1.36 TB) - 0110.5414 (125654 Sec) - DP0180+DP0179
     16:28  ITP:Act/20160512.20:55/2023 lfs-isr1 XenClient XenBatch Zab00‎ diffhist +2,264‎ Rfulcher talk contribs : Last backup did manage to finish before the next weeks jobs: # Refine === Speed up slowest US_Colleague_Pool & US_XCPNG82_Pool tasks by initially writing to faster CS ===

11 April 2024

      20:12  ITP:Act/20160512.20:55/2023 lfs-isr1 XenClient XenBatch Zab00‎‎ 4 changes history +14,498‎ [Rfulcher‎ (4×)]
     
20:12 (cur | prev) +216‎ Rfulcher talk contribs one more vm finished.
     
19:32 (cur | prev) +331‎ Rfulcher talk contribs * Around 18:50 the replication to jbod2 finished. * Then at 19:00 the rebalancing replication to Jupiter appeared to resume. * The lfs-us-isr1-09.digipen.edu disk writes remain at 28~43MBps the whole time.
     
17:50 (cur | prev) +3,289‎ Rfulcher talk contribs Start a loop that will Moving VHDs from ssd1 to raid2 goal, Hourly fix goal for 30 min old VHDs
     
16:25 (cur | prev) +10,662‎ Rfulcher talk contribs 20240405 XenBatch Zab02 Cron === .... .... Record each pools email report subject. # ==== Idea to enhance the XenSnapBackupsAuto.sh to report activity into XCPNG_Center:Alerts ==== # ==== Jupiter Repliation making US_Colleague_Pool & US_XCPNG82_Pool significantly slower
     18:46  ITP:Act/20140425.19:10/20230726 202307RnDAcademy‎ diffhist +3,380‎ Rfulcher talk contribs →‎Missing the English ‎(en)‎ Language: Chatlog: Charles Yacoubian described how he saw many English ‎(_temp_06f62152d846e4c8f7f9d0e825bf5c3d)‎. entries and uninstalled and installed other languages to fix them. 202303-OpenWorld and OpenWorld seem fine.
      18:30  ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks‎‎ 2 changes history +1,868‎ [Rfulcher‎ (2×)]
     
18:30 (cur | prev) +155‎ Rfulcher talk contribs →‎202404 Daily RSync 2015-2024 and DigiPen Moodle: MoodleSG2024 570G 169G 78% # Moodle2024 980G 176G 85%
     
16:44 (cur | prev) +1,713‎ Rfulcher talk contribs →‎202404 Daily RSync 2015-2024 and DigiPen Moodle: lfs-sg-moodle1 MFSMasterP:72.154% # lfs-sg-rcs1 MFSMasterP:66.656% # Both are fine.
     18:28  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 2 changes history +1,488‎ [Rfulcher‎ (2×)]
     
18:28 (cur | prev) +297‎ Rfulcher talk contribs lfs-us-isr1 DIFFERENCE=.4935 # mfs-data (66.49%) # lfs-us-courses (82.18%)
     
17:48 (cur | prev) +1,191‎ Rfulcher talk contribs Jupiter lfs-us-isr DIFFERENCE=.4936 # JBOD mfs-data (62.86%) # lfs-us-courses (79.77%) # At ~16:15 changed Zab goal to ssds1(ssdssc03(lfs-us-isr1-09)) because last weekend backups are still in progress and maybe half finished.
     16:39  ITTR Talk:Storage/Bricks‎ diffhist +743‎ Rfulcher talk contribs europa:megas,2 pass2 ST10000VN0004-1ZD101 ZA26JQ8N SATA ReAlSec:1520 continues
     16:36  ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03‎ diffhist +621‎ Rfulcher talk contribs : 04/11/24, 16:07 - [it-alerts] saturn SyncImapToLizardFS.sh 31%/ 44%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff # : 04/10/24, 21:50 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 355G 92% /zfs/imap01Archive04
     08:10  ITFAQ:Remote licensing‎ diffhist +3‎ Tabatha.tipton talk contribs →‎Keyshot 10 Tag: Visual edit

10 April 2024

     18:28  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎ diffhist +2,660‎ Rfulcher talk contribs Jupiter lfs-us-isr DIFFERENCE=.5431 # JBOD mfs-data (52.40%) increase _REP_LIMIT from 4 to 5 # lfs-us-rcs Finished # lfs-us-courses (72.79%)
     17:50  ITP:Act/20140425.19:10/20230726 202307RnDAcademy‎ diffhist +1,729‎ Rfulcher talk contribs New === Missing the English ‎(en)‎ Language === but maybe Charles installed it?

9 April 2024

      17:06  ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks‎‎ 3 changes history +8,251‎ [Rfulcher‎ (3×)]
     
17:06 (cur | prev) +76‎ Rfulcher talk contribs →‎202403 Daily RSync 2015-2024 and DigiPen Moodle: Drop TODO: ==== 202404 Daily RSync 2015-2024 and DigiPen Moodle ==== heading prefix
     
17:00 (cur | prev) +301‎ Rfulcher talk contribs →‎202403 Daily RSync 2015-2024 and DigiPen Moodle: : 04/09/24, 09:19 - [it-alerts] cron wwwprod89 distancersync 20240409.0015 20240409.0919 50G:96% 96G:91% 126G:90% 399G:72% 175G:77% MoodleSG2024 561G 175G 77% : 04/09/24, 13:27 - [it-alerts] cron wwwprod27 distancersync 20240409.0015 20240409.1327 374G:93% 111G:95% 575G:81% 329G:88% 495G:86% 211G:82% Moodle2024 941G 211G 82%
     
16:58 (cur | prev) +7,874‎ Rfulcher talk contribs →‎202403 Daily RSync 2015-2024 and DigiPen Moodle: execute ===== 20240409 Give 6GB of RAM from lfs-sg-rcs to lfs-sg-moodle masters =====
     16:06  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎ diffhist +1,366‎ Rfulcher talk contribs At noon lfs-us-isr1 DIFFERENCE=.6101 # mfs-data (38.08%) # lfs-us-rcs (54.41%) # lfs-us-courses (60.91%) #
     15:54  ITP:Act/20140425.19:10/20240312 202405Summer Prep‎ diffhist +1,317‎ Rfulcher talk contribs Figured out === Chunk Goal Oddness for US === was using wrong jbod2 goal instead of proper jbods goal.

8 April 2024

      21:07  ITP:Act/20130612.00:19/2020-Sideloaded Arcade Games‎‎ 2 changes history +1,672‎ [Rfulcher‎ (2×)]
     
21:07 (cur | prev) +189‎ Rfulcher talk contribs →‎20240408 Split_Spirit_Setup.exe: /data/webs/arcade/www_arcade$ mv log/production.log log/production.20240408.log 2.0G
     
20:55 (cur | prev) +1,483‎ Rfulcher talk contribs →‎20240404 Smaller Upload now failing: ==== 20240408 Split_Spirit_Setup.exe ====
     19:38  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 2 changes history +9,131‎ [Rfulcher‎ (2×)]
     
19:38 (cur | prev) +1,561‎ Rfulcher talk contribs Increase _REP_LIMIT for mfs-data and lfs-us-rcs
     
18:12 (cur | prev) +7,570‎ Rfulcher talk contribs Progress, Jupiter DIFFERENCE=.6602 # lfsx201:lfs-data-ssd Finished # mfs-data3 (27.15%) # lfs-us-rcs3 (24.38%) # lfs-us-moodle3 (95.71%) # lfs-us-courses3 (51.15%) # Intend to increase the _REP_LIMIT later today.
     19:35  ITTR:Backup/Bacula/Audits‎ diffhist +102‎ Rfulcher talk contribs Incr07CM - ITTR:Backup/Bacula/Audits/US/2024-04-05-Incr - 0213.4848 (222528 Sec) - 3240176024020 (3.24 TB) - 0213.4546 (222346 Sec) - DP0181+DP0180
N    19:34  ITTR:Backup/Bacula/Audits/US/2024-04-05-Incr‎ diffhist +84,949‎ Rfulcher talk contribs Incr07CM - 0213.4848 (222528 Sec) - 3240176024020 (3.24 TB) - 0213.4546 (222346 Sec) - DP0181+DP0180
      19:28  ITP:Act/20160512.20:55/2023 lfs-isr1 XenClient XenBatch Zab00‎‎ 2 changes history +2,901‎ [Rfulcher‎ (2×)]
     
19:28 (cur | prev) +2,209‎ Rfulcher talk contribs new headings.
     
19:27 (cur | prev) +692‎ Rfulcher talk contribs === 20240329 XenBatch Zab01 Cron === .... .... Record each pools email report subject.
     19:20  ITTR Talk:Storage/Bricks‎ diffhist +1,379‎ Rfulcher talk contribs Old Arche mfsbrick.02.12 mfsbrick.02.17 mfsbrick.02.20 out of service. New Arche mfsbrick0212 mfsbrick0217 mfsbrick0220 into service. moreeuropa:megas,2 ZA26JQ8N ReAlSec:1456
      18:46  ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks‎‎ 2 changes history +1,893‎ [Rfulcher‎ (2×)]
     
18:46 (cur | prev) +489‎ Rfulcher talk contribs →‎202403 Daily RSync 2015-2024 and DigiPen Moodle: lfs-sg-moodle1 MFSMasterP:99.595% Might want to adjust some RAM soon?
     
18:41 (cur | prev) +1,404‎ Rfulcher talk contribs →‎202403 Daily RSync 2015-2024 and DigiPen Moodle: : 04/08/24, 08:01 - [it-alerts] cron wwwprod89 distancersync 20240408.0015 20240408.0801 50G:96% 96G:91% 126G:90% 399G:72% 101G:85% MoodleSG2024 535G 101G 85% : 04/08/24, 17:26 - [it-alerts] cron wwwprod27 distancersync 20240408.0015 20240408.1726 374G:93% 111G:95% 575G:81% 329G:88% 494G:86% 122G:89% Moodle2024 930G 122G 89% # Grow both SG and US 2024 by +100GB
     18:05  ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03‎ diffhist +871‎ Rfulcher talk contribs : 04/08/24, 13:55 - [it-alerts] saturn SyncImapToLizardFS.sh 31%/ 43%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff : 04/07/24, 21:09 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 357G 92% /zfs/imap01Archive04
     01:00  ITTR:Singapore Servers/Umbriel‎ diffhist +735‎ Ssulaimi talk contribs Try convert Umbriel bare metal into xcpng VM, to be housed destination on sm24b.

5 April 2024

      21:36  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 2 changes history +2,042‎ [Rfulcher‎ (2×)]
     
21:36 (cur | prev) +580‎ Rfulcher talk contribs DIFFERENCE=.8111 (40.23%) (3.180%) (6.200%) (27.61%) (24.28%)
     
16:00 (cur | prev) +1,462‎ Rfulcher talk contribs Jupiter lfs-us-isr DIFFERENCE=.8250 # JBOD lfs-data-ssd 28.36% # mfs-data 1.820% # lfs-us-rcs 4.350% # lfs-us-moodle 19.84% # lfs-us-courses 21.65%
     19:27  ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks‎‎ 2 changes history +1,091‎ [Rfulcher‎ (2×)]
     
19:27 (cur | prev) +585‎ Rfulcher talk contribs →‎202403 Daily RSync 2015-2024 and DigiPen Moodle: MoodleSG2024 441G 181G 71% # Moodle2024 893G 156G 86%
     
19:25 (cur | prev) +506‎ Rfulcher talk contribs →‎202403 Daily RSync 2015-2024 and DigiPen Moodle: lfs-sg-moodle1 MFSMasterP:97.643%
     15:59  ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03‎ diffhist +414‎ Rfulcher talk contribs : 04/05/24, 14:13 - [it-alerts] saturn SyncImapToLizardFS.sh 31%/ 43%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff : 04/04/24, 21:11 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 364G 92% /zfs/imap01Archive04

4 April 2024

     22:41 Deletion log Rfulcher talk contribs deleted page ITP Talk:Act/20151208.23:57/20240212-More Raid Storage Refinements ‎(Was just a page for printing)
      22:38  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 2 changes history +22,868‎ [Rfulcher‎ (2×)]
     
22:38 (cur | prev) +17,263‎ Rfulcher talk contribs Executed ==== Arche swap out three 10TB for three 24TB Disks ====
     
20:36 (cur | prev) +5,605‎ Rfulcher talk contribs Execute ==== Jupiter new mfsbrick0300 ==== back into service.
     21:13  ITTR Talk:Storage/Bricks‎ diffhist +5,437‎ Rfulcher talk contribs europa:megas,2 pass2 ST10000VN0004-1ZD101 ZA26JQ8N continues to be weak # BSD all upgraded from 13.2-RELEASE-p10 to 13.3-RELEASE-p1 # Jupiter new 67TB mfsbrick0300 went into service.
     18:54  ITP:Act/20130612.00:19/2020-Sideloaded Arcade Games‎ diffhist +1,712‎ Rfulcher talk contribs 20240404 Smaller Upload now failing
 m   13:11  ITTR:Moodle‎ diffhist +13‎ B.maryott talk contribs →‎Attendance Auditors

3 April 2024

      21:28  ITTR:Moodle/SG-2024Summer‎‎ 2 changes history +119‎ [Pleilani‎ (2×)]
     
21:28 (cur | prev) +34‎ Pleilani talk contribs updated Tag: Visual edit
     
21:20 (cur | prev) +85‎ Pleilani talk contribs assignment role : non-editing Teacher to Xiao Shi, Joanna and Jiamin Tag: Visual edit
     20:59  ITP:Act/20151208.23:57/20240212-More Raid Storage Refinements‎‎ 2 changes history +16,313‎ [Rfulcher‎ (2×)]
     
20:59 (cur | prev) +14,608‎ Rfulcher talk contribs Execute ==== Jupiter Verify MLAG Bond ==== updates to move to active-active lacp bond with jumbo frames MTU 9000
     
18:07 (cur | prev) +1,705‎ Rfulcher talk contribs Reduce ~91% back down to ~85% with lfs-us-isr1-03:~# GoalChangePerAge 7 # Laid out Plans for Near Term Capacity Upgrades:
     19:58  ITP:Act/20161026.02:11/20220718-imap01 imap01Archive03‎ diffhist +414‎ Rfulcher talk contribs : 04/03/24, 11:13 - [it-alerts] saturn SyncImapToLizardFS.sh 31%/ 43%local 21%archive 44%alumni 49%student 66%faculty 29%wanic 75%staff : 04/02/24, 21:16 - [it-alerts] imap01 CompressImapBackups.sh 4.0T 371G 92% /zfs/imap01Archive04
     17:32  ITP:Act/20151208.23:57/20220110-lfs-us-moodle-LoopBacks‎ diffhist +1,199‎ Rfulcher talk contribs →‎202403 Daily RSync 2015-2024 and DigiPen Moodle: : 04/03/24, 07:29 - [it-alerts] cron wwwprod89 distancersync 20240403.0015 20240403.0729 50G:96% 96G:91% 126G:90% 404G:72% 211G:66% MoodleSG2024 406G 211G 66% : 04/03/24, 07:34 - [it-alerts] cron wwwprod27 distancersync 20240403.0015 20240403.0734 374G:93% 111G:95% 575G:81% 329G:88% 484G:86% 175G:84% Moodle2024 873G 175G 84%

2 April 2024

 m   14:50  ITTR:Moodle‎ diffhist +9‎ B.maryott talk contribs Removed David Van Der Linden
Retrieved from "https://inside.digipen.edu/main/Special:RecentChanges"