Server Restart
Server restart required on the 02/05/2018 at 9:00am Down Time 10 min

Update: 12/01/2018 09:00

Hello,

Maintenance Notice: Please be advised that the following Linux hosting server will be undergoing a maintenance for the reason mentioned below.

Reason: Cloud Linux Kernal Investigating


Affected Servers:

server.cpanelireland.com


Maintenance Window:

Date & Time: 13th to the 14th Jan 2018

Duration: There will be intermittent outages through out the investigation

Affected Services: Website - Mail - FTP - SSH - Billling Acounts, Cpanel Ireland Website



Update: 14/01/2018 22:39

We have found a problem with the Cloud linux server's Kernel which prevents the server to boot up. We have contact Cloud Linux and they will be providing a fix for us tomorrow.
Once this fix has been applied. we will be rebooting the server for the changes to take effect, the down time will be 5 - 10 Minutes.

Please Check back here again to find out the time of the reboot.


Update: 15/01/2018 21:00

The Investigation is still continuing, the server did not need a restart today, The procedure will continue tomorrow.
Please check back here for updates.



Update: 28/01/2018 24:00

for updates on the RHEL/CentOS 6 kernel breaking XenServer issue.
https://access.redhat.com/solutions/3312501

RHEL 6 XEN guest with kernel-2.6.32-696.18.7.el6.x86_64 hangs on boot

Issue

• RHEL 6 guest doesn't boot with kernel-2.6.32-696.18.7.el6.x86_64 after patching.
• System boots up with previous kernels successfully.

• The guest hangs on the following messages at boot:

  Raw

  (XEN) mm.c:2560:d27 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn 21ed59b (pfn 1e05)
  (XEN) d27:v0: unhandled page fault (ec=0000)
  (XEN) Pagetable walk from ffffffffff400000:
  (XEN)  L4[0x1ff] = 000000214d50f067 0000000000001a91
  (XEN)  L3[0x1ff] = 000000214d50e067 0000000000001a92
  (XEN)  L2[0x1fa] = 0000000000000000 ffffffffffffffff
  (XEN) domain_crash_sync called from entry.S: fault at ffff82d080203a7a entry.o#create_bounce_frame+0x125/0x13b
  (XEN) Domain 27 (vcpu#0) crashed on cpu#0:
  (XEN) ----[ Xen-4.4.4OVM  x86_64  debug=n  Not tainted ]----
  (XEN) CPU:    0
  (XEN) RIP:    e033:[]
  (XEN) RFLAGS: 0000000000000202   EM: 1   CONTEXT: pv guest (d27v0)
  (XEN) rax: 0000000000000000   rbx: ffffffffff400000   rcx: 0000000000000004
  (XEN) rdx: 0000000000000010   rsi: ffffffffff400000   rdi: ffffffff81a03e68
  (XEN) rbp: ffffffff81a03e48   rsp: ffffffff81a03e00   r8:  0000000000000000
  (XEN) r9:  0000000000007ff0   r10: 0000000000000000   r11: 0000000000000000
  (XEN) r12: ffffffff81a03e58   r13: ffffffffff400000   r14: ffffffffff410000
  (XEN) r15: ffffffff81a03e68   cr0: 000000008005003b   cr4: 00000000001526f0
  (XEN) cr3: 000000214d512000   cr2: ffffffffff400000
  (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
  
  
  
  
  
  

Update: 14/02/2018 18:48

New Kernel Released, A compatible version of this kernel for Xen-PV will go to beta release at the end of the week.

Changelog since kernel-3.10.0-714.10.2.lve1.5.9:

• AL-2579: fixed GRUB checks when system uses UEFI;
• CLKRN-153: fixed possible race between dentry_kill and RCU-walk;
• CLKRN-203: fixed kernel panic;
• CLKRN-210: fixed CVE-2017-5715 [branch target injection] also known as 'Spectre Variant 2';
• fixed SPEC_CTRL and IBPB_SUPPORT features setting after microcode update;
• KMODLVE-123: postpone lvp pmem limit enforcement if first attempt failed;
• KMODLVE-149: check a notifier pointer while unregistering notifiers;
• CLEN-61: restored serialized access to TSC through mfence;
• restored patch to fix regression and cover CLKRN-172.

 
Update: 21/02/2018 15:11

Finally An updated Kernel has been released to address the problem with the server rebooting.
The below Kernel Has been installed and the server has rebooted Successfully.


CloudLinux 6 kernel version 2.6.32-896.16.1.lve1.4.53 is available for download from our updates-testing repository.

Changelog since kernel-2.6.32-896.16.1.lve1.4.52:

• CLKRN-219: fixed boot on Xen PV instances;
• CKSIX-148: disabled certain printk's for UBC OOM path to not spam into dmesg log.

Hello,

We found a network issue in shared server server63 and we are working at the moment to fix it. We will provide more information regarding the situation soon.

Update Oct 1st 21:02

Our admins are still working on this situation. Server is up but not accessible via public network. We will keep you posted about the situation.

Update Oct 1st 23:47

Engineers are still working on resolving access to server63. We are currently working as fast as possible to bring the server back on-line.

Update Oct 2nd 00:12

We have been unable to recover the server as the file system has been corrupted. We have started a restore of the server and this will take approx 7 hours to complete. We apologise for any inconvenience caused

Ransomware 

At 5:30am this morning we were notified by our data centre that Ransomware had been injected and replicating through our network.
All our servers run on Linux Operating systems so the Ransomeware did not affect any of the files or user data stored on the servers but could have infected any website users using windows operating systems.
With this information a decision was made to take all servers offline until a security audit was completed and all disks where scanned and all threats were removed.

Several Patches have been installed on our network to minimise the risk of this attack again, but the truth of it is that these attackers can and will alter their code to try these attacks again.
The attack was not targeted at Cpanel Ireland directly but was spread through other networks across the country.

All Servers are now back on-line and rest assured that no information held on the Cpanel Ireland Network was retrieved or compromised.

A Denial Of Service Attack was detected today on our network at 14:00pm Today 06/02/2017.
The attack was migrated and and service was restored at 14:15 Today 06/02/2017.

The attack originated from Argentina and the netherlands from the following ip address, 189.124.18.250 - 218.255.06.106 - 79.129.38.181

Root Kit Attack 


Tough Medicine For Windigo Victims

“The Ebury backdoor deployed by the Windigo cybercrime operation does not exploit a vulnerability in Linux or OpenSSH,” continued Léveillé. “Instead it is manually installed by a malicious attacker. The fact that they have managed to do this on tens of thousands of different servers is chilling. While anti-virus and two factor authentication is common on the desktop, it is rarely used to protect servers, making them vulnerable to credential stealing and easy malware deployment.”

The Advice given was - If sysadmins discover their systems are infected, they are advised to wipe affected computers and reinstall the operating system and software.  It is essential that fresh passwords and private keys are used, as the existing credentials must be considered compromised.

For a higher level of protection in future, technology such as two-factor authentication should be considered.

“We realise that wiping your server and starting again from scratch is tough medicine, but if hackers have stolen or cracked your administrator credentials and had remote access to your servers, you cannot take any risks,” explains Léveillé.  “Sadly, some of the victims we have been in touch with know that they are infected, but have done nothing to clean up their systems – potentially putting more internet users in the firing line.”

All computer users are reminded that they should never reuse or choose easy-to-crack passwords.

No Hosting accounts were breached in this attack ONLY the OS was effected.

Steps taken to clean server.

1. Format all Drive's
2. Install OS
3. update OS
4. Install Cpanel
5. Update Cpanel
6. Configure OS & Cpanel
7. Install Firwall / Update & Configure
8. Upload Hosting Account backups from 31/12/2014
9. Restore Backups
10. Check account Restoration for errors

Currently running Centos 6.6 x86_64 xenpv Server

The Network infrastructure will be upgraded over the weekend This is a necessary upgrade to the data center to facilitate growing demands on the system.

Nothing to report